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FOREWORD 


A  concern  of  the  U.S.  Army  Intelligence  Center  and  School  (USAICS)  is  the 
development  of  methods  for  evaluating  military  intelligence  (MI).  The  Army 
uses  a  number  of  methods  to  evaluate  soldier  performance,  including  the  Skill 
Qualification  Test  (SQT)  and  the  Army  Training  and  Evaluation  Program  (ARTEP). 
These  methods,  however,  are  designed  to  evaluate  generalized  soldier  skills 
rather  than  tasks  and  processes  specific  to  MI.  The  current  effort  will  de¬ 
velop  a  method  for  evaluating  the  functional  effectiveness  of  MI  that  will 
complement  the  SQT  and  ARTEP.  The  method  will  consist  of  two  components:  an 
instrument  that  will  assess  the  output  of  MI  in  terms  of  its  utility  to  intel¬ 
ligence  users  and  a  fault  diagnosis  technique  that  will  trace  failures  in  the 
output  to  causes  within  the  MI  process.  The  report  describes  the  development 
of  the  first  component. 

This  research  was  conducted  within  the  Fort  Huachuca  Field  Unit  of  the 
U.S.  Army  Research  Institute  under  a  contract  to  Science  Applications  Inter¬ 
national  Corporation.  The  project  was  sponsored  by  the  Directorate  of  Train¬ 
ing  and  Doctrine,  U.S.  Army  Intelligence  Center  and  School  in  a  Letter  of 
Agreement  dated  14  March  1986.  The  results  of  the  research  were  provided  to 
the  sponsor;  the  Directorate  of  Combat  Developments,  USAICS;  and  the  Com¬ 
mander,  U.S.  Army  Intelligence  Agency.  When  completed,  the  measurement  tool 
will  be  used  in  tactical  training  exercises. 
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MEASUREMENT  AND  EVALUATION  OF  MILITARY  INTELLIGENCE  PERFORMANCE 


EXECUTIVE  SUMMARY 


Requirement : 

The  objective  of  this  research  is  to  develop  and  validate  a  measurement 
method  for  evaluating  the  effectiveness  of  Military  Intelligence  (MI).  The 
research  is  a  3-year  effort.  The  first  year  of  the  research  is  the  subject  of 
this  report. 

Year  1  of  this  research  effort  focused  on  bringing  the  perspective  of  the 
intelligence  user  into  the  evaluation  of  MI.  During  Year  1,  ve  identified  the 
range  of  issues  associated  with  the  assessment  of  MI  and  developed  a  basis  for 
a  measurement  instrument  using  multi-attribute  utility  assessment  (MAUA). 


Procedure: 

A  variety  of  data  collection  techniques  were  used  during  the  first-year 
research.  Techniques  included  individual  interviews,  multi-attribute  utility 
assessment  (MAUA),  and  survey  questionnaires. 

Individual  interviews  were  conducted  for  the  following  reasons:  (1)  to 
identify  and  define  factors  associated  with  an  intelligence  user's  view  of 
outputs  provided  by  intelligence,  (2)  to  identify  sources  of  variation  in  a 
user’s  utility  assessment,  and  (3)  to  identify  discrete  elements  of  informa¬ 
tion  content  needed  by  intelligence  users  and  the  sources  of  variation  in 
these  needs. 

MAUA  was  used  to  quantify  intelligence  user  utility.  The  procedure 
followed  the  basic  MAUA  implementation  steps  and  completed  the  following  in 
Year  1:  (1)  defined  the  decision  problem,  (2)  identified  and  ordered  the 

factors  to  be  used  in  evaluating  alternatives,  and  (3)  elicited  relative 
importance  weights  for  elements  and  nodes  in  the  hierarchical  order. 

A  survey  questionnaire  was  developed  and  used  to  assess  military  intel¬ 
ligence  products.  Objectives  of  the  survey  were  the  following:  (1)  to  de¬ 
termine  whether  an  intelligence  product  orientation  was  too  narrow  a  focus  in 
this  research,  and  (2)  to  assess  and  compare  the  perspective  of  users  regard¬ 
ing  doctrinal  and  nondoctrinal  intelligence  products. 


Findings: 

From  the  first  year  of  research  we  have  concluded  that  the  effectiveness 
of  MI  is  directly  related  to  how  useful  the  MI  output  is  to  the  user.  Assess¬ 
ing  the  effectiveness  of  MI  without  consideration  of  the  user  utility  of  in¬ 
telligence  outputs  is  too  narrow  a  focus. 
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The  application  of  MAUA  in  this  effort  produced  a  hierarchy  of  user  in¬ 
formation  needs  that  (1)  reflects  the  needs  of  different  types  of  users  (e.g., 
G3f  G2,  G4,  FSC),  (2)  can  be  weighted  by  users  to  reflect  the  relative  impor¬ 
tance  of  the  information  items,  and  (3)  discriminates  information  needs  of 
various  contexts  (e.g.,  peacetime,  wartime). 

The  survey  of  intelligence  products  provided  two  major  conclusions. 

First,  a  product  orientation  as  defined  by  intelligence  doctrine  is  not  suf¬ 
ficient  for  this  research.  Second,  intelligence  users  recognize  and  assign 
high  utility  to  many  intelligence  outputs  that  are  not  identified  by  intel¬ 
ligence  doctrine  as  products.  In  light  of  the  survey  findings,  the  general 
term  "output"  was  adopted  to  connote  any  output  from  intelligence  to  the  user. 


Utilization  of  Findings: 

The  information  needs  hierarchy  can  be  viewed  as  a  data  base  of  informa¬ 
tion  needs  to  which  user  utility  can  be  applied.  We  see  two  potential  appli¬ 
cations  for  the  data  base.  One  application  is  the  development  of  profiles  for 
classroom  evaluation  of  student -generated  output.  A  profile  would  essentially 
be  a  weighted  data  base  of  information  items  represented  by  the  information 
needs  hierarchy.  This  would  permit  instructors  to  employ  a  user-specific, 
situationally  dependent  profile  to  represent  a  "mythical  G3"  against  which 
student-generated  intelligence  output  could  be  assessed.  A  second  application 
of  the  data  base  of  information  needs  is  for  assessing  the  performance  of  MI 
effectiveness  in  a  dynamic  setting  such  as  a  field  exercise.  Both  applica¬ 
tions  of  the  data  base  require  the  development  of  a  scoring  approach,  one 
topic  of  research  for  the  second  year  of  this  research  effort. 

The  second  year  of  this  research  effort  will  be  concerned  with  the  devel¬ 
opment  of  the  process  corollary  to  user  utility.  A  major  future  direction  for 
this  research  is  to  trace  user  utility  into  the  intelligence  process  in  an 
effort  to  identify  the  sources  of  low  user  utility  and  develop  remedies  for 
utility  shortfalls. 

The  third  year  of  this  research  effort  will  be  concerned  with  validation 
of  both  the  measurement  instrument  and  the  diagnostic  procedure  for  tracing 
user  utility  into  the  intelligence  process. 
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MEASUREMENT  AND  EVALUATION  OF  MILITARY  INTELLIGENCE 

PERFORMANCE 

INTRODUCTION 


Overview 

This  report  presents  the  technical  progress  for  die  first  year  research  of  the 
three-year  research  program  entitled  "Measurement  and  Evaluation  of  Military 
Intelligence  (MI)  Unit  Information  Processing  Performance."  The  program  began 
in  September  1986  under  the  sponsorship  of  the  U.  S.  Army  Research  Institute  and 
the  U.  S.  Army  Intelligence  Center  and  School.  The  research  under  this  project  is 
being  performed  by  Science  Applications  International  Corporation  (S  AIC)  with 
suppon  from  George  Mason  University. 

The  intent  of  this  research  effort  is  to  develop  a  methodology  for  evaluating 
military  intelligence  effectiveness.  This  report  describes  the  problem,  our 
approach,  results,  and  future  plans.  It  is  one  of  two  concerning  this  research  effort. 
A  supplemental  report  is  available  that  is  oriented  toward  the  military  audience  of 
this  research.  The  following  report  is  designed  primarily  for  an  audience  interested 
in  the  technical  rather  than  military  aspects  of  the  research. 

The  Army  Need 

Military  intelligence  (MI)  units  and  staff  elements  have  several  instruments 
available  to  them  for  measuring  individual  and  unit  performance.  The  Skill 
Qualification  Test  (SQT)  measures  individual  task/skill  competency  for  a  given 
Military  Occupational  Specialty  (MOS).  Similarly,  the  Army  Training  and 
Evaluation  Program  (ARTEP)  provides  standards  for  unit  task/skill  competency. 
Rom  the  MI  perspective,  the  combination  of  the  S QT  and  ARTEP  provides  some 
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measure  of  perfoimance  of  an  MI  unit,  but  not  the  complete  picture.  As  shown  in 
Figure  1,  there  is  a  missing  component  of  effectiveness  evaluation  that  assesses  the 
complete  functionality  of  the  MI  infrastructure. 

This  research  effort  intends  to  fill  in  that  missing  component  in  the 
effectiveness  equation  with  a  measurement  instrument  that  allows  MI  to  assess  its 
functional  effectiveness.  We  are  not  developing  an  approach  to  assess  an  individual 
intelligence  analyst  -  that  is  the  domain  of  the  SQT.  Nor  will  we  duplicate  unit  task 
competency  measurement  as  specified  in  the  ARTEP. 

The  Problem 

Possibly  the  most  difficult  problem  challenging  researchers  in  this  effort  is 
the  determination  of  what  constitutes  effectiveness  for  ML  To  some  people  the 
ultimate  measure  of  MI  effectiveness  is  whether  a  war  is  won  or  lost  From  a 
measurement  standpoint,  isolating  the  MI  contribution  to  mission  success  is 
exceedingly  difficult  to  measure  due  to  a  large  number  of  confounding  variables. 

An  individual  who  is  currently  serving  as  a  division  Assistant  Chief  of  Staff  for 
Intelligence  phrased  his  view  of  MI  effectiveness  in  this  manner 

Intelligence  reporting  is  not  rote  and  the  success  of  intelligence  reporting 
is  not  measured  by  pages  produced  but  rather  in  lives  saved  or  lost. 

Good  analysts  and  good  analysis  is  H2I  pontification  but  a  usable, 
credible  product.. 

W  .;j  the  tide  of  this  study  concerns  MI  unit  effectiveness,  our  approach  is 
mon  ...  lately  described  as  a  measurement  of  MI  ineffectiveness.  We  are 
attempting  isolate  the  faults  in  MI,  not  the  strengths.  Further,  we  are  looking  at 
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SQTs 


ARTEPs  Measurement 

Instrument 


Figure  1.  Measurement  of  MI  unit  effectiveness. 
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the  faults  from  the  perspective  of  those  supported  by  MI,  those  who  rely  on 
intelligence  to  provide  diem  with  information  about  the  opposing  force  and 
battlefield  area. 


Objective 

The  objective  of  this  research  effort  is  to  develop  and  validate  an  instrument 
suitable  for  evaluating  the  performance  of  a  tactical  MI  "unit"  which  includes  the 
essential  staff  supporting  elements.  During  the  first  year  of  the  research  we  sought 
to  identify  the  range  of  issues  associated  with  the  assessment  of  MI,  focus  the 
research  for  the  instrument  development,  and  develop  and  apply  an  approach  to  the 
instrument  development 
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APPROACH 


Overview 

Year  1  of  this  research  effort  focused  on  the  development  of  an 
understanding  of  intelligence  output  utility  from  die  perspective  of  the  intelligence 
user.  Measuring  die  utility  of  MI  outputs  involves  understanding  a  complex  set  of 
factors.  Any  single  technique  would  run  the  risk  of  failing  to  identify  some  key 
factors  and  developing  only  a  partial  understanding  of  the  problem.  To  avoid 
missing  potentially  important  factors  this  project  employed  a  variety  of  data 
collection  techniques,  including:  individual  interviews,  Multi-Attribute  Utility 
Assessment  (MAUA),  and  survey  questionnaires.  These  techniques  were  used 
along  with  a  supporting  research  base  developed  by  the  project  team.  Figure  2 
provides  an  overview  of  the  multifaceted  approach  to  Year  1  of  this  research  effort. 

Each  technique  employed  in  this  research  effort  had  specific  objectives.  The 
use  of  individual  interviews  had  three  primary  objectives:  (1)  identify  and  define 
those  factors  associated  with  a  user's  view  of  the  utility  of  outputs  provided  from 
intelligence,  (2)  identify  sources  of  variation  in  a  user’s  utility  assessment,  and  (3) 
identify  discrete  elements  of  information  content  needed  by  intelligence  users  and 
the  sources  of  variation  in  these  needs.  The  technique  of  MAUA  was  directed 
toward  the  development  of  a  metric  to  be  used  in  the  training  environment  to  assess 
intelligence  products.  A  questionnaire  about  the  use  of  the  military  intelligence 
products  focused  on  two  objectives:  (1)  determining  whether  an  intelligence 
product  orientation  in  the  study  was  too  narrowly  focused;  and  (2)  assessing  and 
comparing  the  perspective  of  users  regarding  doctrinal  and  non-doctrinal 
intelligence  products. 
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Figure  2.  Overview  of  Year  1  research  approach. 
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The  following  sections  discuss  each  technique  used  in  this  research. 
Additionally,  the  advantages  and  disadvantages  of  each  technique  are  identified. 

A  Proposed  Utility  Framework  for  Evaluation  of  MI  Effectiveness 
This  research  developed  an  orientation  toward  utility  as  the  underlying  base 
of  MI  effectiveness  from  a  simple  model  of  intelligence.  MI  produces  intelligence, 
puts  the  intelligence  into  a  variety  of  possible  forms,  and  gives  the  intelligence  to  a 
user.  The  question  is,  how  do  you  measure  the  effectiveness  of  what  the  unit  is 
doing?  The  measurement  problem  converged  on  the  concept  of  utility  because  it 
can  be  related  to  the  intelligence  process,  to  output  forms,  and  to  users.  The 
intelligence  process  is  concerned  with  utility  because  of  the  need  to  focus  scarce 
resources  (analysts,  time,  sensors,  and  sources)  in  a  way  that  produces  high  utility 
output.  Users  want  and  need  high  utility  outputs  from  intelligence  and  want  to 
minimize  time  and  effort  to  use  those  outputs. 

Early  in  our  research,  it  became  apparent  that  the  study  of  MI  effectiveness 
was  a  question  of  "When  is  what  MI  produces  useful  to  the  user?"  The  user 
perspective  had  to  be  included  to  address  MI  effectiveness.  It  also  became  apparent 
that  this  restatement  of  the  problem  reflected  a  complex  series  of  questions:  "What 
does  MI  produce?;  What  factors  make  it  useful?  Who  are  the  users?  Over  what 
period  do  you  measure  performance?"  A  utility  framework  was  selected  for 
looking  at  MI  effectiveness  because  it  provided  far  quantifying  subjective  notions 
of  usefulness  as  perceived  by  the  user.  Utility  also  dealt  with  a  qualitative  and 
quantiative  view  of  information  and  relevance.  Our  utility  framework  identified 
three  areas  of  focus  as  illustrated  in  Figure  3. 
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Potential  Study  Focus  Potential  Study  Focus  Potential  Study  Focus 


Assess  fulfillment  of 
requirements 

Measure  use  of  resources 

Assess  how  information 
triage  is  accomplished 

Assess  analysis 

Assess  dissemination 


Assess  SINGLE  output 
in  qualitative  or 
quanta  tive  manner 

Assess  MULTIPLE 
outputs  in  qualitative  or 
quantitative  manner 


Influences  on  user 

assessment  of  output: 

-  Factors  external  to 
outputs 

-  Factors  internal  to  user 

-  Factors  associated  with 
a  single  output 

-  Factors  associated  with 
multiple  outputs 


Figure  3.  A  proposed  utility  framework  for  evaluation  of  MI  effectiveness. 
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Year  1  of  this  research  has  focused  on  developing  the  user  perspective  of 
the  utility  of  intelligence  outputs  and  developing  a  methodology  for  bringing  that 
perspective  into  a  measurement  instrument  to  apply  to  outputs.  Primarily,  this 
study  focused  on  the  factors  that  influence  a  user's  assessment  of  intelligence 
output.  Factors  studied  included:  those  external  to  the  user,  those  associated  with  a 
single  output,  and  those  associated  with  multiple  outputs.  Factors  internal  to  the 
user  are  concerned  with  biases  including:  organizational  biases,  personal  biases, 
and  cognitive  biases.  Internal  factors,  although  representing  viable  influences  on 
user  assessment,  were  considered  outside  the  scope  of  this  research. 

Individual  Interviews 

The  purpose  of  the  individual  interviews  in  this  effort  was  to  bring  the 
perspective  of  the  intelligence  users  into  the  research.  Interviews  provide  a  rich 
source  of  information  and  a  means  for  identifying  the  general  range  of  issues 
involved  in  the  problem.  Interviews  with  subject  matter  experts  (SMEs)  provide  a 
way  of  building  a  basic  conceptual  framework,  which  can  be  refined  in  subsequent 
interviews.  Interviews  provide  a  good  method  for  developing  an  initial 
understanding  of  the  problem  and  structure  of  the  concept  space,  without  undue 
influence  of  preconceived  notions  of  the  researchers.  Anecdotal  information 
obtained  in  interviews  helps  to  characterize  the  nature  of  a  problem  and  allows 
analogies  to  be  drawn  to  other  domains.  Finally,  interviews  provide  a  good  check 
to  ensure  that  important  issues  are  not  overlooked. 

There  are  several  disadvantages  to  individual  interviews.  For  one, 
interviews  are  relatively  time  consuming  to  conduct  and  analyze.  Typically, 
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interviews  are  conducted  with  a  relatively  small  sample  of  SMEs,  raising  a  question 
about  the  generalization  of  the  findings.  The  results  of  interviews  are  generally 
described  in  qualitative  terms  that  do  not  lend  themselves  well  to  the  development  of 
a  rigorously  quantitative  metric.  While  individual  interviews  provide  valuable 
insights  into  the  problem,  the  interviews  alone  cannot  develop  the  type  of 
quantitative  metric  desired  from  this  project 

Interview  Population 

Active-duty  Army  staff  officers  appeared  to  be  the  optimal  SME  target  but 
constraints  on  their  availability  was  a  factor  in  limiting  their  use.  For  this  reason, 
individual  interviews  were  conducted  with  in-house  personnel  who  recently  retired 
from  an  Army  operational  assignment  to  supplement  interviews  with  active-duty 
personnel. 

Eight  individuals  within  S  AIC  were  identified  who  had  recently  retired  from 
the  Army  with  operations  backgrounds.  The  experience  of  these  individuals 
included  corps,  division,  and  brigade  echelons  serving  in  the  roles  of  commander, 
G3,  assistant  G3,  and  S3.  Primary  specialties  of  the  interview  participants 
included:  infantry,  field  artillery,  armor,  combat  engineering,  chemical,  and  combat 
communications.  The  advantage  of  using  in-house  participants  included:  (1)  time 
constraints  were  not  placed  on  our  use  of  these  individuals,  and  interview  sessions 
could  be  conducted  for  significantly  longer  periods  of  time  than  possible  with 
active-duty  Army  staff  officers;  and  (2)  clarification  questions  and  followup 
interviews  were  easily  accomodated  using  in-house  personnel. 
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The  research  team  was  given  access  to  numerous  active-duty  Army  staff 
officers  from  various  continental  U.S.  (CONUS)  divisions.  Various  sites  were 
visited  and  individuals  currently  serving  as  division  staff  officers.  Personnel 
interviewed  included  the  following  division  staff  functions:  G3,  G3Plans,  G30ps, 
Chemical,  FSE  (Fire  Support  Element),  G2  (Intelligence),  and  G2Plans. 

Protocol  Pcvglpprrent 

The  purpose  of  individual  interviews  was  to  identify  the  range  of  concerns 
of  operations  users  in  regard  to  intelligence  products.  Typically,  the  results  of  one 
set  of  interviews  served  as  a  guide  to  establishing  new  objectives  for  the  next  round 
of  interviews.  In  this  manner,  the  interview  protocols  varied  with  the  objectives  of 
the  interviews.  Table  1  provides  an  overview  of  the  objectives  for  each  series  of 
interviews.  While  interview  objectives  differed,  the  approach  to  individual 
interviews  followed  a  consistent  pattern:  draft  protocol,  pre-test;  conduct 
interviews,  analyze  data,  and  revise  protocol  for  next  interview  set. 

In-house  interviews  were  conducted  in  two  phases.  The  first  phase  of 
interviews  consisted  of  open-ended  questions  looking  into  a  range  of  general  issues 
related  to  an  operations  user's  view  of  intelligence  outputs.  In  particular,  the 
interviews  sought  to  identify  a  user's  information  needs  for  specific  contexts,  such 
as  peacetime  and  active  hostilities,  in  an  attempt  to  determine  if  information  needs 
were  context  independent  or  context  dependent  In  addition,  interview  participants 
were  queried  to  identify  factors  associated  with  good  intelligence.  These  factors 
were  used  in  the  second  phase  of  in-house  interviews  when  participants  were  asked 
to  rate  both  information  types  and  factors.  Appendix  A  contains  the  protocols  used 
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TABLE  1.  Individual  Interviews:  Summary  of  Objectives 


INTERVIEW  LOCATION 

OBJECTIVE 

In-House 

Phase  I 

Identify  factors  that  influence  the  usefulness 
fo  intelligence  outputs. 

Identify  user  information  requirements  in 
two  different  situational  contexts. 

Elicit  opinions  regarding  ways  to  improve 
intelligence. 

Phase  II 

Obtain  importance  ratings  of  factors  that 
influence  the  usefulness  of  intelligence 
outputs  for  different  situational  contexts. 

Obtain  importance  ratings  of  information 
types  in  different  situational  contexts. 

Site  Visits 

Fort  Bragg 

Elicit  important  user  tasks  performed  and 
associated  information  requirements  from 
intelligence. 

Elicit  factors  that  influence  information  needs. 

Elicit  factors  that  influence  user  confidence 
in  information/intelligence. 

Fort  Stewart 

Identify  changes  in  information  needs  related 
to  situation  context  and  user  task. 

Identify  factors  that  influence  the  usefulness 
of  intelligence. 

Fort  Lewis 

Confirm  previously  identified  factors  that 
influence  user  assessment  of  the  utility  of 
intelligence  outputs. 

Obtain  definitions  of  utility  factors. 
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for  the  in-house  interviews.  In-house  interviews  were  conducted  at  various  times 
throughout  the  first  year  effort. 

Site  visit  interviews  were  conducted  at  three  Forces  Command 
(FORSCOM)  division  locations.  At  division  staff  officers  were  interviewed 
individually.  The  first  site  visit,  to  the  82nd  Airborne  Division  at  Fort  Bragg,  used 
a  scenario-based  protocol  in  which  user  tasks  and  their  respective  information  needs 
from  intelligence  were  identified.  Additionally,  participants  interviewed  were  asked 
to  identify  factors  that  influence  their  confidence  in  the  information  they  receive  as 
well  as  factors  that  influence  the  kind  of  information  needed. 

The  second  site  visit  was  made  to  the  24th  Infantry  Division  (Mechanized) 
at  Fort  Stewart.  The  protocol  differed  substantially  from  the  one  used  at  Fort 
Bragg.  The  major  difference  in  this  protocol  was  that  it  required  recognition  rather 
than  recall  on  the  part  of  interview  participants.  Participants  were  given  a  task  list 
(prepared  in  advance  for  each  individual  role)  as  well  as  information  item  lists.  The 
task  and  information  lists  were  both  developed  from  Army  doctrinal  material. 
During  the  interview,  participants  were  asked  to  rank  their  tasks  by  importance  and 
identify  their  information  needs  for  important  tasks.  The  information  item  list  was 
reviewed  by  participants  for  completeness  and  accuracy.  Participants  were  given 
two  contexts  in  which  to  focus  their  responses  to  our  questions:  current  day-to-day 
job  (peacetime/noncombat)  and  hostilities.  Contexts  and  tasks  were  varied  in  an 
effort  to  identify  changes  in  information  needs.  Questions  were  also  asked  of 
participants  in  an  effort  to  identify  factors  they  associate  with  the  usefulness  of 
intelligence  outputs. 
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The  third  site  visit  was  made  to  the  9th  Infantry  Division  at  Fort  Lewis.  On 
the  basis  of  the  previous  interviews,  the  protocol  developed  for  this  site  visit  was  a 
significant  modification  over  those  previously  used.  The  objective  of  the  interviews 
at  Fort  Lewis  was  to  obtain  definitions  of  utility  attributes  previously  identified. 

The  reason  for  setting  such  an  objective  stemmed  from  our  realization  that 
consistent  definitions  were  a  necessity.  Prior  to  the  site  visit,  tentative  definitions 
were  developed  for  each  of  the  utility  attributes.  The  bases  of  the  tentative 
definitions  were  dictionary  definitions.  The  definitions  were  supplemented  with 
examples  for  each  of  the  attributes.  Four  basic  questions  were  asked  about  each  of 
the  attributes:  (1)  Did  the  definition  proposed  capture  the  interview  participant’s 
meaning  of  the  attribute  (and  to  redefine  as  needed)?;  (2)  Was  the  attribute  a  factor 
in  assessing  the  usefulness  of  an  intelligence  output?;  (3)  Could  a  good  and  bad 
example  of  the  attribute  be  given?;  and  (4)  Does  peacetime  or  wartime  environment 
change  the  participant's  standard  or  meaning  of  the  attribute?.  A  number  of  open- 
ended  questions  was  included  in  the  protocol  to  be  asked  if  time  permitted. 

Interview  ITwwtorc 

Our  approach  to  these  interviews  was  to  develop  the  protocol,  administer 
the  protocol,  analyze  the  results,  and  modify  the  protocol  for  the  next  site  visit  In 
this  manner,  the  site  visits  allowed  us  to  explore  a  variety  of  topics  with  active 
Army  division  staff  officers. 

All  interviews  were  conducted  by  trained  psychologists  with  extensive 
experience  in  interviewing.  Participants  were  informed  of  the  purpose  of  the 
research  effort  and  the  interview  as  well  as  assured  of  the  confidentiality  of  their 
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responses.  With  the  permission  of  each  interview  participant,  interviews  were  tape 
recorded  to  provide  verbatim  transcripts  for  purposes  of  analysis.  Additionally, 
notes  were  taken  by  the  investigator  to  highlight  major  points  discussed  during  the 
interviews. 


Multi-Attribute  Utility  Assessment 

MAUA  is  a  structured  approach  to  decision-making  designed  to  assess  how 
much  value  or  utility  decision-makers  place  on  the  various  alternatives  open  to  them 
on  the  basis  of  (a)  the  relative  priority  of  the  factors  they  consider  important,  and 
(b)  how  well  each  alternative  scores  on  these  factors.  There  are  five  basic  steps  in 
implementing  MAUA: 

Step  1.  Define  the  decision  problem  in  terms  identifying  the  decision¬ 
makers,  the  decision  alternatives,  and  those  situational  factors  that  affect  the 
decision,  but  arc  beyond  (or  external  to)  the  decision-maker's  control. 

Step  2.  Identify  the  factors  to  be  used  in  evaluating  the  alternatives.  These 
factors  arc  organized  into  a  hierarchy  where  similar  factors  arc  grouped  together. 
The  top-level  factors  in  the  hierarchy  typically  represent  the  broad  evaluation  factors 
the  decision-maker  is  trying  to  use  when  evaluating  the  alternatives.  Each  top-level 
factor  is  further  subdivided  until  they  are  well  enough  defined  so  that  attributes  can 
be  scored. 

Step  3.  Assign  relative  weights  to  the  factors  and  attributes  in  the  hierarchy 
to  represent  the  relative  importance  (or  priority)  of  the  different  decision  factors. 

Step  4.  Evaluate  an  actual  set  of  alternatives  by  assigning  a  utility  score  to 
each  of  the  bottom-level  attributes  in  the  hierarchy  for  each  alternative. 
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Step  5.  Combine  the  relative  weights  and  scores  to  obtain  an  overall  utility 
score  for  each  alternative. 

MAUA  is  typically  used  to  achieve  two  goals:  to  better  understand  the 
decision-maker's  subjective,  decision-making  process;  and  to  provide  an  explicit, 
quantitative  approach  to  evaluating  alternatives.  Both  MAUA  goals  were  consistent 
with  the  project's  goals  of  understanding  how  users  evaluate  MI  outputs  and  of 
developing  instruments  for  measuring  the  utility  of  MI  outputs,  respectively. 

MAUA  provides  a  rigorous  and  quantitative  approach  to  understanding  how 
different  factors  influence  product  utility.  The  strongest  advantage  of  this  approach 
involve  the  conceptual  rigor  of  the  MAUA  model.  It  develops  a  consistent, 
detailed,  highly-structured,  and  quantitative  representation  of  the  problem.  This 
type  of  representation  is  internally  consistent  and  mathematically  powerful.  It 
provides  a  powerful  analytic  tool  for  examining  utility. 

Another  advantage  of  MAUA  is  sensitivity  analysis.  Specifically,  the 
consistent  quantitative  structure  resulting  from  a  MAUA  permits  quantitative 
assessment  of  how  much  difference  in  the  relative  weights  and/or  scores  are 
required  to  change  the  conclusion  (e.g.,  that  Alternative  A  is  better  than  Alternative 
B). 

MAUA  can  be  effectively  used  in  domains  where  there  are  a  small  sample  of 
SMEs,  or  where  access  to  SMEs  is  constrained  such  as  in  this  effort  The 
generalization  of  obtained  results  can  be  improved  with  access  to  more  SMEs,  if 
possible  within  time  and  cost  constraints. 

From  a  pragmatic  perspective,  the  MAUA  approach  to  instrument 
development  offers  the  advantage  of  a  fast,  cost-effective  developmental  cycle;  e.g.. 
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although  MAUA  requires  substantial  time  from  participating  SMEs,  the  reduced 
sample  requirements  allow  for  faster,  less  expensive  instrument  development  than 
in  a  survey  approach. 

There  are,  however,  some  difficulties  to  the  MAUA  approach.  The  simplest 
approach  to  implementing  a  MAUA  assumes  independence  between  criteria  and 
between  attributes  within  criteria.  Independence  is  typically  defined  by  saying  that 
the  value  of  one  criterion  is  not  necessarily  affected  by  the  value  of  another 
criterion;  the  same  holds  for  attributes  within  criteria.  People  tend  to  object  to  the 
independence  assumption  initially,  but  typically  a  structure  can  be  created  that 
maintains  the  independence  assumption.  However,  if  independence  cannot  be 
maintained,  more  complicated  combination  rules  are  required. 

The  initial  development  of  the  hierarchy  is  subjective  in  nature. 

» 

Specifically,  team  researchers  work  with  a  small  group  of  SMEs  for  rather  lengthy 
working  sessions  in  an  effort  to  develop  a  hierarchy  that  receives  consensus 
approval  from  the  SMEs.  Refinements  of  the  hierarchy  may  be  obtained  by 
presenting  the  subjective  hierarchy  to  other  groups  of  SMEs.  There  has  been 
minimal  empirical  research  evaluating  the  relative  accuracy  of  different  MAUA 
structuring  techniques.  The  research  of  Adelman,  Sticha,  and  Donnell  (1986) 
indicated  that  "top-down"  and  "bottom-up"  MAUA  structuring  techniques  resulted, 
on  the  average,  in  different  looking  but  equally  accurate  MAUA  hierarchies. 

Alternatives  to  the  subjective  development  of  the  hierarchy,  such  as 
statistically  based  approaches,  are  not  viable.  Factor  analysis  and  its 
multidimensional  scaling  have  been  proposed  as  alternative  methods  for  obtaining 
the  hierarchy,  but  they  are  seldom  (if  ever)  used  outside  of  marketing  research. 
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The  MAUA  approach  to  the  problem  of  quantifying  intelligence  user  utility 
was  selected  because  of:  (1)  the  viability  of  MAUA  as  a  method  for  quantifying 
utility;  (2)  the  MAUA  results  led  themselves  to  statistical  procedures  for  validation 
purposes;  and  3)  the  cost  effective  nature  of  the  approach. 

The  initial  steps  taken  in  applying  MAUA  to  the  measurement  and 
evaluation  of  Ml  output  utility  were  accomplished  through  a  decision  conference 
with  active-duty  Army  personnel.  The  conference  was  conducted  at  Army 
Research  Institute’s  field  office  at  Fort  Leavenworth,  Kansas,  with  four  senior- 
level  personnel  representing  operations  (G3),  logistics  (G4),  fire  support  element 
(FSE),  and  intelligence  (G2).  The  conference  focused  on  understanding/assessing 
the  utility  of  different  items  of  information  for  various  users  under  different 
situational  contexts. 

In  an  effort  to  prepare  the  Army  participants  for  the  conference,  a  package 
of  "read-ahead"  materials  was  sent  to  the  participants  prior  to  the  conference.  These 
materials  can  be  found  in  Appendix  B.  Briefly,  the  package  contained  a 
"strawman"  hierarchy  of  information  items  representing  needs  and  a  description  of 
the  situational  factors  that  the  project  team  hypothesized  were  important  to 
measuring  the  utility  (or  relevance)  of  information  items  contained  in  the  hierarchy. 
A  MAUA  hierarchy  was  developed  reflecting  information  items  representative  of 
intelligence  outputs.  The  information  needs  hierarchy  included  54  discrete, 
independent  items  which  related  to  typical  intelligence  topic  groupings.  The  goals 
of  the  conference  were  to  (a)  obtain  consensus  cm  a  hierarchy  of  information  items 
as  a  framework  for  organizing  information  needs;  (b)  obtain  consensus  on  a  set  of 
situational  factors  for  defining  situational  context;  and  (c)  obtain  importance  weights 
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testing  the  hypothesis  that  the  relative  value  (i.e.,  utility  or  relevance)  of  different 
types  of  information  varies  by  users  under  differing  situational  contexts.  All  three 
goals  were  achieved  during  the  conference.  The  results  of  the  conference  may  be 
found  in  the  Results  section  of  this  report.  The  following  discussion  briefly 
describes  the  development  of  the  materials  for  the  read-ahead  package  for  the 
conference. 


Situational  Context 

Situational  context  was  identified  as  a  major  source  of  variability  in 
information  needs  and  possibly  intelligence  output  utility.  Situational  context  could 
mean:  who  was  using  it,  when  was  it  being  used  (e.g.,  peacetime  or  wartime), 
where,  and  for  what  purpose.  These  factors  are  essentially  not  under  the  control  of 
an  individual  user,  therefore  are  external  to  an  individual  and  pan  of  the  total 
situational  context 

The  notion  of  situational  context  appeared  as  a  basic  premise  in  the 
formulation  of  all  Army  doctrinal  literature,  offering  additional  confirmation  of  its 
important  role  in  output  utility.  Doctrinal  literature  habitually  implied  that  there  are 
situational  variables  that  may  be  the  driving  factors  for  accomplishing  intelligence 
tasks.  To  date,  the  literature  has  stopped  short  of  suggesting  that  the  utility  of 
different  elements  of  information  may  also  vary  in  differing  situations. 

Given  that  situational  context  could  directly  impact  on  how  a  user  assesses 
information  needs  or  assesses  the  utility  of  intelligence  outputs,  it  was  imperative  to 
identify  the  relevant  variables  in  context  Doctrinal  literature  provided  a  basis  for  the 
identification  of  those  external  factors  that  combine  to  form  a  situational  context. 
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Table  2  contains  the  first  strawman  set  of  external  factors.  Each  column  heading 
from  left  to  right  represents  a  major  situational  component  The  elements  in  each 
row  are  the  descriptive  variations  applicable  to  each  heading. 

Level  of  Conflict  includes  possibilities  intended  to  distinguish  among  a 
range  of  operational  activity  states,  each  felt  to  have  unique  attributes  for  potential 
impact  on  information  utility. 

‘Lie  User  category  attempted  to  account  for  different  specific  functional  uses 
of  intelligence  by  arraying  the  major  staff  and  command  elements  with  unique 
intelligence  requirements. 

User  Echelon,  because  of  inherently  different  operational  responsibilities 
ranging  from  tactical  maneuver  orientation  to  one  of  strategic  planning,  accounted 
for  a  significant  situational  variable. 

Operational  State  is  a  refinement  of  levels  of  conflict  into  discrete  states 
relative  to  tactical  operations. 

Application  is  a  direct  derivative  of  traditional  operational  staff  activities. 
Each  level  represents  a  singularly  exclusive  activity  felt  to  have  a  potentially  unique 
view  of  utility  for  intelligence  outputs. 

The  most  doctrinally  direct  situational  component.  Mission,  represents  those 
prescribed  responsibilities  placed  on  a  subordinate  command  by  a  senior 
headquarters. 

A  situational  context  was  defined  by  selecting  an  element  from  each 
column.  The  number  of  potentially  different  situations  was  unmanageably  large  for 
this  methodology.  Not  all  situations  were  necessarily  different  from  the  perspective 
of  intelligence  output  Understanding  how  much  influence  each  situational  factor 
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TABLE  2.  External  Factors  Matrix. 


LEVEL  OF  CONFLICT 

USER 

ECHELON 

OPERATIONAL 

STATE 

APPLICATION 

MISSION 

Peacetime/Noncombat 

G1 

National 

Status  Quo 

Data  Base/ 
Record  Keeping 

Offense 

Terrorism 

G2 

U&S 

Command 

Pre-Hostility 

Protection/ 

Security 

Defense 

Condgency  Operations 

G3 

Theater 

Hostility 

Planning 

Retrograde 

Low-Intensity 

Conflict 

G4 

Corps 

Post-Hostility 

Execution 

Special 

Operations 

Mid-Intensity 

Conflict 

G5 

Division 

Contingency 

Operations 

High-Intensity 

Conflict 

CDR 

Brigade 

Joint/ 

Combined 

FSE 

Battalion 

2 


had  in  discriminating  intelligence  output,  became  an  agenda  for  the  MAUA  decision 
conference  in  order  to  minimize  the  number  of  situational  factors  that  had  to  be 
considered. 


Strawman  Hierarchy 

A  strawman  hierarchy  of  information  items  was  prepared  for  the  decision 
conference  and  contained  in  the  read-ahead  package  for  the  participants.  The  actual 
information  items  contained  in  the  hierarchy  originated  from  an  extensive 
information  item  list  developed  for  the  individual  interviews  conducted  at  Fort 
Stewart.  This  strawman  was  developed  from  numerous  doctrinal  materials  and 
confirmed  by  active-duty  personnel  interviewed  to  be  all-inclusive  of  information 
potentially  needed  by  users.  The  hierarchical  organization  of  the  information  items 
was  also  founded  in  doctrine.  Specifically,  the  intelligence  product  called  an 
Intelligence  Estimate  was  used  to  guide  the  organization.  This  product  was  selected 
because  of  the  breadth  of  information  items  contained  in  an  estimate.  The  entire 
strawman  hierarchy  of  information  needs  may  be  found  in  Appendix  B.  Figure  4 
contains  the  two  highest  levels  of  the  strawman  hierarchy  used  in  the  Decision 
Conference  at  Fort  Leavenworth. 

A  product  orientation  was  used  to  develop  the  first  strawman  hierarchy  of 
information  needs,  as  discussed  above.  However,  the  orientation  in  our  research 
focused  on  intelligence  outputs  rather  than  products.  The  reason  for  this  shift  in 
perspective  was  based  on  our  research  finding  that  there  are  numerous  viable 
outputs  from  intelligence  that  do  not  carry  the  formal  label  of  "product.”  The  use  of 
the  term  ”output"  means  anything  and  everything  that  is  produced  by  intelligence 
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L  Reference  to  Relevance 
of  Information 


A.  Refers  to  mission 

B.  Refers  to  commander's  guidance 


II.  Description  of  Battlefield 
Area 


A.  Weather 

B.  Terrain 

C  Other  characteristics  of  area 


m.  Description  of  Enemy 
Situation 


A.  Enemy  disposition  and  composition  described 

B.  Strength  of  enemy  fore  described 

C  Recent  and  present  significant  activities 
described 

D.  Peculiarities  and  weaknesses  described 

E.  Special  targets  identified 


IV.  Description  of  Enemy 
Capabilities 


A.  Enemy  Courses  of  Action  (COAs)  enumerated 

B.  Changes  in  enemy  situation  summarized 
C  Changes  in  battlefield  area  identified 

D.  Enemy  capabilities  summarized 

E.  Enemy  COAs  analyzed  and  discussed 


V.  Conclusions  in  Terms  of 


A.  Effects  of  intelligence  considerations  on 
operations 

B.  Effects  of  (he  area  of  operations  on  friendly 
COAs 

C  Probable  enemy  COAs 
D.  Enemy  vulnerabilities 


VI.  Other  Factors 


A.  Gaps  in  information  (and  their  implications) 

B.  Inconsistencies  with  previous  information 
identified 

G  Specificity  of  geographic  locations 


Figure  4.  Strawman  information  needs  hierarchy  (levels  1  and  2  only). 
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for  a  user.  The  research  evidence  regarding  outputs  and  products  are  discussed  in 
greater  detail  later  in  this  report. 

Decision  Conference 

The  read-ahead  package  was  sent  to  all  conference  participants  two  weeks 
prior  to  the  conference.  In  addition,  the  package  contained  the  agenda  and 
objectives  for  the  conference  as  well  as  background  information  concerning  the 
research  effort  The  time  allotted  for  the  conference  was  an  entire  day,  which  was 
required  to  achieve  all  the  desired  objectives. 

MI  Product  Usage  Questionnaire 

A  questionnaire  approach  was  developed  early  in  the  research  to  survey 
users  about  intelligence  product  usage,  similarities  and  differences  in  users,  and 
potential  problems  to  investigate.  The  term  "intelligence  product"  is  supposed  to 
distinguish  the  finished  result  of  the  intelligence  process  (e.g.,  analysis, 
confirmation  from  other  sources)  from  the  raw  data,  information,  (and  noise) 
vacuumed  into  the  input  side  of  the  intelligence  system.  Intelligence  focuses  on 
supporting  the  command  staff  with  information,  frequently  in  the  form  of  written 
and  verbal  products.  The  development  of  products  requires  tasking  and 
management  of  significant  resources  by  intelligence  producers,  presumably  in 
proportion  to  the  importance  of  the  product.  Users  in  turn  must  provide  resources 
to  read  the  product,  attend  a  briefing,  or  extract  information  from  a  product 
Doctrine  identifies  a  large  number  of  products  and  defines  the  content,  format  and 
a  standard  reporting  cycle.  While  doctrine  defines  numerous  parameters  of 
products,  we  questioned  whether  doctrine  covered  all  the  outputs  received  by 
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intelligence  users.  If,  in  fact,  doctrine  defined  "all  there  was,"  then  doctrine  could 
be  used  as  a  yardstick  for  the  evaluation  of  intelligence.  In  other  words,  the  metric 
for  evaluation  could  be  doctrinally  based. 

Our  questionnaire  sought  to  determine  whether  a  strict  intelligence  doctrinal 
definition  of  products  was  sufficient  for  our  research  effort.  The  development  of 
die  questionnaire  was  guided  by  two  constraints:  (1)  time  to  complete  the 
questionnaire  should  be  no  more  than  30  minutes;  and  (2)  the  questionnaire  should 
be  sufficiently  self-explanatory  so  that  a  project  researcher  was  not  required  to 
administer  the  survey. 


Selection  of  Products  for  Questionnaire 
The  first  step  in  the  questionnaire  development  was  to  identify  products  to 
be  included  in  the  questionnaire.  We  began  by  identifying  intelligence  products 
recognized  by  intelligence  doctrine,  Field  Manual  34-3, 1986,  and  Field  Manual 
34-1, 1984,  were  the  prime  sources  of  the  study  in  the  phase.  Intelligence  doctrine 
identifies  numerous  written  products  and  various  types  of  formal  briefings  as  the 
output  of  intelligence.  In  addition,  doctrine  identifies  the  various  formats  that  some 
products  may  take  under  specific  situations  (e.g..  Intelligence  Annex  if  issued 
separate  from  an  Operations  Order,  Estimate  for  Stability  Operations,  Intelligence 
Summary  for  Joint  Operations).  The  count  of  products  becomes  quite  long  when 
consideration  is  given  to  the  permutations  of  products  created  by  format  and 
specific  tactical  operations. 

In  an  effort  to  comply  with  the  directed  constraints,  we  elected  not  to 
include  every  product  or  intelligence  output  in  our  questionnaire.  To  limit  the  set  of 
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products,  we  eliminated  any  products  not  produced  by  division  intelligence.  In  the 
case  of  four  different  imagery  reports,  these  were  included  as  a  group.  In  addition, 
we  eliminated  some  products  dial  survey  parti  rip  ants  might  rarely  receive, 
particularly  in  the  peacetime  environment,  such  as  nuclear  biological  and  chemical 
products  (Le.,  NBC-1,  NBC-2,  etc.).  However,  we  did  allocate  space  on  the 
questionnaire  for  users  to  add  products  that  they  felt  should  be  included  in  the 
analysis.  The  final  version  of  the  survey  contained  ten  doctrinal  products.  Two 
types  of  briefings  were  included  in  the  final  questionnaire  version,  formal  and 
informal.  Much  of  the  information  exchange  between  intelligence  and  users  takes 
place  in  an  informal  manner.  The  six  non-doctnnal  products  selected  for  inclusion 
in  the  questionnaire  were  not  specified  by  doctrine  as  products.  Rather,  these 
"products"  are  related  to  the  Intelligence  Preparation  of  the  Battlefield  (IPB) 
process.  They  were  identified  by  in-house  SMEs  for  inclusion  in  this 
questionnaire.  One  point  of  interest  to  this  study  was  whether  or  not  operations 
users  would  have  familiarity  with  these  "products"  and  further  whether  or  not  they 
would  view  them  as  products.  The  products  included  in  the  questionnaire  may  be 
found  in  Table  3. 


Questionnaire  Construction 

Given  the  objective  of  this  survey,  there  were  numerous  topics  that  the 
survey  could  potentially  cover.  A  totally  open-ended  set  of  questions  was  not 
seriously  considered  as  a  vehicle  for  a  number  of  reasons;  for  one,  such 
questionnaires  take  too  much  time  to  fill  out  and  may  be  prone  to  interpretation 
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TABLE  3.  Military  Intelligence  Product  Usage  Questionnaire: 

k 

Products  Surveyed. 


Intelligence  Summary  (INTSUM) 

Periodic  Intelligence  Report  (PERINTREP) 
Supplementary  Intelligence  Report  (SUPINTREP) 
Intelligence  Annex 
Intelligence  Estimate 
Intelligence  Report  (INTREP) 

Situation  Report  (SITREP) 

Imagery  Reports 
Spot  Report 

Responses  to  Information  Requests 

Bri-gfines; 

Formal  Briefings 
Informal  Briefings 


W  WiH 


Collection  Priorities 
Named  Areas  of  Interest  (NAIs) 
Target  Areas  of  Interest  (TAIs) 
Target  List 
Target  Nomination 
Target  Priority  List 
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problems.  Our  approach  was  to  identify  a  small  number  of  topics  to  address  that 
could  be  presented  in  a  highly  structured  manner  to  ease  respondent  completion  and 
interpretation. 

The  product  usage  questionnaire  contained  three  sections.  The  first  section 
asked  subjects  to  supply  demographic  data  about  themselves.  The  second  section 
asked  subjects  to  rate  all  products  on  specific  topics.  The  third  secdon  contained 
open-ended  questions.  Appendix  C  contains  the  final  version  of  the  survey 
questionnaire. 

Demographic  Data.  Collection  of  demographic  data  was  necessary  to 
determine  if  the  survey  participants  were  in  fact  the  appropriate  target  group  for  this 
survey.  The  intent  of  the  survey  was  to  assess  the  usefulness  of  intelligence 
products  from  the  perspective  of  operational  users.  Additional  demographic 
information  collected  included  rank,  primary  and  alternate  specialties,  and  a  brief 
identification  of  their  division  level  operational  staff  experience.  Individuals 
participating  in  this  survey  included  active  Army  personnel  interviewed  at  the 
various  site  visited  and  decision  conference  attendees  as  well  as  other  active  Army 
personnel  discussed  in  the  Results  section  of  this  report. 

Questionnaire  Topics.  The  structured  questionnaire  addressed  four  topics: 
familiarity,  frequency  of  use,  usefulness,  reason  for  use.  Each  topic  contained  a 
brief  instruction  paragraph  and  a  list  of  products  by  the  scale  or  rating  categories  to 
be  completed.  Etch  form  allowed  for  the  addition  of  products  at  the  bottom  of  the 
form.  Consistent  five  point  or  category  assessment  was  used  in  an  attempt  to  ease 
the  completion  of  the  questionnaire.  Products  were  listed  in  the  same  order  for  each 
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rating  form;  doctrinal  products  were  listed  first,  followed  by  briefings,  and  non- 
doc  triii  al  products. 

Fust,  all  subjects  were  asked  to  rate  their  familiarity  with  each  intelligence 
product  contained  in  the  survey.  Since  we  wanted  to  obtain  the  opinion  of  users 
about  products  based  on  their  experience  with  the  products,  this  created  a  need  to 
assess  a  respondent's  experience  with  each  product  A  five-point  scale  was 
utilized  with  the  following  meanings:  1  -  Not  at  All  Familiar,  2  -  Slightly  Familiar, 
3  -  Moderately  Familiar,  4  -  Very  Familiar,  and  5  -  Extremely  Familiar.  Subjects 
were  asked  to  select  only  one  category  to  describe  how  well  acquainted  they  were 
with  a  particular  product  The  importance  of  the  familiarity  rating  was  to  tell  us 
whether  or  not  subsequent  ratings  were  based  on  the  first-hand  experience  of  the 
subject  If  a  subject  rated  his  familiarity  with  a  particular  product  as  "not  at  all 
familiar,"  then  answers  to  subsequent  questions  about  that  product  would  not  have 
actual  experience  as  a  basis. 

The  second  topic  addressed  in  the  questionnaire  concerned  the  frequency  of 
use  of  products.  The  questionnaire  contained  five  categories:  Don't  Use,  File  for 
Later  Use,  Use  Once,  Use  Repeatedly,  and  Use  Right  Away.  Forced  choice 
among  these  categories  did  not  appear  appropriate  from  discussions  with  in-house 
SMEs.  Thus,  subjects  were  instructed  to  select  as  many  as  necessary  to  describe 
their  frequency  of  use  of  a  product 

The  third  topic  addressed  in  the  questionnaire  dealt  with  product  usefulness. 
A  five-point  scale  was  developed  on  which  usefulness  was  to  be  rated:  1  -  Not 
Useful,  2  -  Slightly  Useful,  3  -  Moderately  Useful,  4  -  Very  Useful;  and 
5  -  Extremely  Useful.  Subjects  were  instructed  to  consider  how  well  a  product  fit 
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their  requirements  or  the  sufficiency  of  the  product  to  meet  their  intelligence  needs. 
Subjects  were  required  to  make  a  single  usefulness  category  selection  for  each 
subject. 

A  fourth  topic  was  added  to  the  questionnaire  after  the  initial  data  collection 
from  five  subjects.  This  topic  dealt  with  the  reason  for  use  of  each  intelligence 
product.  Subjects  were  asked  to  characterize  the  reason(s)  why  they  use  a 
particular  product  via  the  following:  Only  Source  Available,  Best  Source  Available, 
Easy  to  Use,  Most  Timely  Information,  and  other  reasons  that  they  were  asked  to 
specify. 

Open-Ended  Questions.  The  final  portion  of  the  survey  instrument 
contained  a  number  of  open-ended  questions.  Subjects  were  given  the  option  of 
answering  any  or  all  of  these  questions  at  their  discretion.  The  open-ended 
questions  were  included  in  the  survey  to  elicit  user  opinions  not  captured  by  the 
structured  portion  of  the  survey.  Subjects  were  given  the  opportunity  to 
recommend  modifications  to  specific  intelligence  products  in  the  first  open-ended 
question,  identify  experience  at  specific  commands  or  unit  assignments  that  formed 
the  basis  of  their  assessment  of  products  in  the  earlier  portion  of  the  questionnaire, 
and  identify  assumptions  that  they  made  about  the  intelligence  products  they  rated. 
Also,  subjects  were  asked  what  questions  should  have  been  asked  in  the  survey  that 
were  not.  The  concluding  question  asked  subjects  what  they  would  like  to  point 
out  to  researchers  looking  at  the  operational  perspective  of  intelligence  products. 
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Questionnaire  Evaluation 

SMEs  were  consulted  to  determine  the  appropriate  ordering  of  products. 
Pilot  testing  of  the  questionnaire  was  accomplished  with  in-house  volunteers  not 
familiar  with  the  research  effort  to  determine  the  time  for  completion,  adequacy  of 
instructions,  and  general  comments  on  the  survey  instrument  All  members  of  the 
research  team  including  government  representatives  offered  guidance  and 
suggestions  concerning  the  survey  instrument 
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RESULTS 


Summary 

The  first  year  of  this  research  led  us  to  the  major  conclusion  that  the 
effectiveness  of  MI  is  directly  related  to  how  useful  its  output  is  to  users.  While  the 
effectiveness  of  MI  may  be  looked  at  from  an  internal  perspective,  such  as  the 
effective  use  of  resources,  resource  usage  is  too  narrow  a  focus  forjudging 
effectiveness.  By  themselves,  internal  resources,  even  if  effectively  used,  do  not 
indicate  whether  the  intelligence  output  was  useful  to  the  user  or  how  the  use  of 
resources  may  be  contributing  to  the  mission.  A  broader  view  of  MI  effectiveness 
considers  the  added  value  of  inference  contributed  by  the  human  component  and 
the  added  value  of  processing  that  makes  the  output  more  useful  to  users.  Equating 
MI  effectiveness  to  user  utility  allows  us  to  proceed  with  the  development  of  a 
measurement  instrument  that  focuses  on  quantitative  assessment  of  utility  factors. 

The  results  of  the  MI  product  usage  questionnaire  (MI  PUQ)  led  us  to 
conclude  that  using  a  product  orientation  as  defined  by  intelligence  doctrine  to 
develop  a  measurement  instrument  is  inadequate.  Users  of  intelligence  recognize 
and  assign  high  utility  to  many  intelligence  outputs  that  are  not  recognized  as 
products  by  intelligence  doctrine.  The  more  general  term  "outputs"  was  adopted  by 
this  effort  rather  than  "product"  to  connote  any  output  from  intelligence  to  a  user. 

The  results  of  the  MAUA  conference  challenge  some  aspects  of  intelligence 
doctrine.  The  strawman  information  needs  hierarchy  presented  at  the  conference 
was  organized  around  the  doctrinal  intelligence  product  called  an  estimate,  selected 
because  of  the  breadth  of  information  contained  in  the  estimate.  Participants  in  the 
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conference  reorganized  the  information  needs  to  appear  more  like  the  outputs  of  the 
XPB,  a  doctrinal  intelligence  process  but  not  a  doctrinal  product 

Hie  results  of  the  activities  undertaken  during  the  first  year  of  the  research 
effort  are  organized  by  technique  in  the  order  of  interviews,  MAUA  session,  and 
questionnaire. 


Results  of  Individual  Interviews 

The  individual  interviews  were  conducted  in  an  effort  to  address  multiple 
objectives  identifying:  (1 )  operations  user  information  needs,  (2)  the  range  of 
factors  that  operations  users  consider  in  the  assessment  of  the  utility  of  an 
intelligence  output,  and  (3)  any  factors  external  to  an  operations  user  that  influence 
either  his  or  her  information  needs  or  utility  factors. 

Individual  interviews  were  conducted  with  two  populations  of  military 
personnel.  One  population  was  drawn  from  the  range  of  in-house  operations 
expertise  made  available  to  the  research  team,  in  which  two  separate  phases  of 
in-house  interviews  were  conducted,  comprised  of  eight  individuals  participants. 
The  second  population  consisted  of  individuals  currently  serving  as  division  staff 
officers,  which  consisted  of  fourteen  active  Army  division  staff  officers  at  three 
different  site  locations.  The  following  discussion  of  the  results  of  the  interviews  is 
organized  by  topics  addressed  by  the  interviews. 

Information  Needs 

The  identification  of  the  information  needs  of  operations  users  was  a  major 
focus  of  the  open-ended  interviews.  For  most  of  the  interviews,  the  identification 
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of  information  needs  was  conducted  in  open-ended  interviews.  In  this  manner 
participants  were  placed  in  a  position  of  recalling  their  needs  from  memory.  While 
this  approach  yielded  many  user  information  needs,  it  was  also  quite  time 
consuming.  For  that  reason,  we  modified  the  procedure  for  one  of  the  later  site 
visits  and  prepared  a  lengthy  list  of  information  items  to  permit,  though  not  require, 
interview  participants  to  identify  their  needs  from  the  list.  Additionally,  participants 
were  asked  to  modify  the  list  if  necessary  in  an  attempt  to  compile  a  comprehensive 
list  of  possible  information  needs. 

The  strawman  information  item  list  was  developed  from  doctrinal  material 
as  well  as  Army  studies  relating  to  information  requirements  generally  associated 
with  the  division  echelon.  This  list  included  information  items  elicited  from  the 
open-ended  interviews  as  well.  Active  Army  division  staff  officers  who  reviewed 
the  list  found  it  to  be  complete  and  offered  no  suggestions  for  further  modifications. 
The  information  item  list  may  be  found  in  Figure  5. 

Influences  on  Information  Needs 

An  initial  hypothesis  in  this  research  was  that  information  needs  are  likely  to 
vary  with  the  task  an  individual  is  performing.  This  hypothesis  proved  to  be  untrue 
based  on  numerous  individual  interviews.  One  G3,  for  example,  maintained  that 
his  information  needs  were  influenced  by  a  context  that  did  not  include  his  task  at 
hand  be  die  task  "coordinating  all  aspects  of  maneuver"  or  "developing  the 
command  SOP."  In  a  sense,  it  would  appear  that  information  needs  of  a  user 
represent  the  elements  of  the  user's  mental  model  of  the  battlefield.  Further,  it  is 
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Enemy  Capabilities  (Who,  What) 
unit  identification 

unidentified  units  and  available  information 
unit  descriptions  and  disposition 
force  composition  and  organization 
unit  missions 

other  OB  (personalities,  history,  tactics,  training,  etc.) 
rime/distancc  factors 

capability  to  hit  high  value  friendly  locations 
ability  to  influence  friendly  scheme  of  maneuver 
relative  combat  power 

strength^weakniesses  in  logisticaVtechnical  capabilities 

Enemy  Disposition  (Where) 
forward  trace 
unit  locations 
main  area  of  attack 
echelonmcnt 
support  elements 
reserves 
staging  areas 
supply  lines 
supply  areas 
air  corridors 

Enemy  Strengths  and  Weaknesses  (in  What  Strength) 
unit  status 
resupply  rates 
critical  nodes 
main  force 
non-effective  units 
morale 

Enemy  Intentions 
concentrations  of  forces 
preparations 
readiness  levels 
courses  of  action 

most  probable  course  of  action  (units  and  strengths  in 
main  attack) 

enemy  intelligence  collection  priorities 

Enemy  Activity  (When) 
phase  lines  on  enemy  avenues  of  approach 
activities  in  rear  area 
activities  in  front  area 
activities  in  deep  area 
relevant  activity  outside  area  of  influence 
activities  organized  functionally  (air,  nuc,  chem,  naval, 
logistic,  REC,  etc.) 

movements  (combat  maneuver,  artillery,  airborne, 
logistics) 

Battlefield  Area 
about  movement  corridors 
about  specific  areas  of  interest 
target  areas 
boundaries 

general  terrain  analysis 
general  weather  analysis 


Targets 

target  folders  (doctrinal  templates,  imagery, 

weaponcering,  etc.) 

high  value  target  lists 

target  nominations 

potential  nuclear  targets 

enemy  concentrations 

bomb  damage  asscssement 

EW  targets 

special  targets 

prohibited  tar  gets/areas 

OPSEC/Fricndly  Vulnerabilities 
enemy  intelligence  capabilities 
friendly  vulnerabilities 
espionage,  sabotage,  Spematz,  airborne,  etc. 
RECacuvities 

Information  Requirements 

list  of  requirements  by  prioritiesAequestor 
intelligence  assets  tasked  to  requirements 
status  of  fulfilling  requirements  with  collection 
plan 

information  gaps  and  explanation 
sufficiency  of  intelligence  assets  to  meet 
requirements 

Intelligence  Summaries 

historical  record  of  significant  events 
comparison  of  prediction  to  actual  events 
projections  of  significant  enemy  actions 

Intelligence  Credibility  for  Projections 

technical  capabilities/limitations  of  data  sources 
continuity  of  tracking  critical  enemy  force 
elements 

completeness  of  situation  display 
how  much  of  enemy  mass  is  represented  by 
units  being  tracked 
intelligence  asset  status 

Changes 

OB  changes  (character  of  enemy) 
movements 

unusual  activities,  unanticipated  events 
serendipitous  events 
changes  in  strengths/weaknesses 
disposition  of  units 

replacement,  rotation,  reinforcement,  or 

resupply  of  units 

enemy  intelligence  activities 

advance  to  new  phase  line 

enemy  COA  change 

indicators  of  decision  points 

situation  changes  (weather,  terrain,  friendly) 


Figure  5.  Information  item  list 
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die  model  and  its  associated  gaps  that  drive  information  needs  rather  than  tasks 
driving  information  needs. 


During  in-house  individual  interviews,  participants  were  asked  to  rate  the 
importance  of  individual  information  items  in  relation  to  two  different  contexts, 
peacetime  and  wartime.  The  histogram  contained  in  Figure  6  shows  the  mean 
importance  rating  of  the  information  items  in  the  peacetime  and  wartime  context 
For  most  items,  the  importance  rating  was  higher  in  the  active  hostilities  than  in 
peacetime  (the  one  exception  was  political  climate).  The  means  and  standard 
deviations  of  the  importance  ratings  may  be  found  in  Appendix  D. 

Intelligence  Output  Utility  Factors 

The  individual  interviews  sought  to  identify  the  range  of  factors  that 
influence  operations  users'  assessment  of  the  utility  of  an  intelligence  output.  The 
in-house  interviews  sought  to  query  interview  participants  in  depth  about  the  factors 
and  their  importance  in  particular  contexts.  Site  interviews,  on  the  other  hand, 
restricted  queries  to  the  identification  of  the  factors. 

Individual  interviews  identified  a  total  of  twenty-one  possible  factors  that 
influence  the  utility  assessment  of  an  intelligence  output.  These  factors  are  shown 
in  Table  4.  The  order  reflects  the  frequency  of  responses  of  the  factors.  Timeliness 
and  accuracy  were  the  most  frequently  given  responses  to  a  question  about  factors 
that  influence  the  utility  of  intelligence  outputs. 
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moderately 

important 


HOT 
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AT  AU 


INFORMATION  ITEMS 


Figure  6.  Importance  rating  of  information  items 
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TABLE  4.  Intelligence  Output  Utility  Factors. 


Timeliness 

Accuracy 

Currency 

Relevant  to  division  mission 

Reliability  (of  source,  analyst,  or  information) 

Credibility  (of  source  or  analyst) 

Detail 

Summary 

Filtered 

Processed 

Complete 

Interpreted 

Concise 

Gear 

Simple 

Information 

Intelligence 

Use  of  graphics 

Source  confirmed 

Source  identification 

Confidence  of  producer  in  output 
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The  factors  contained  in  Table  4  are  the  actual  terms  used  by  the  interview 
respondents,  elicited  by  open-ended  questions.  Examination  of  the  list  of  factors 
revealed  apparent  redundancy.  For  example,  the  terms  "unambiguous"  and  "clear" 
arc  synonymous.  In  an  effort  to  identify  redundancy,  definitions  were  developed 
for  each  factor.  Standard  dictionary  definitions  were  used  as  the  basis.  In  addition, 
transcripts  of  the  interviews  were  reviewed  to  identify  instances  where  the  use  of  a 
term  appeared  to  differ  with  the  interview  respondents’  intended  meaning.  The 
utility  factor  "accuracy"  is  a  particular  case  in  point  Accuracy,  in  general  usage, 
refers  to  correctness,  or  quality  of  representing  reality.  Some  users  appear  to  use 
the  term  accuracy  to  mean  precision  (e.g.,  eight-digit  grid  coordinates  are  more 
precise  than  six  digits  though  the  accuracy  may  be  equal).  The  utility  factors 
"timeliness”  and  "currency"  both  appeared  to  relate  to  the  user’s  need  to  receive 
information  "in  time"  to  make  use  of  it  Timeliness,  on  the  one  hand,  appeared  to 
relate  to  the  overall  output  while  currency  appeared  to  reflect  the  age  of  information 
content  Perishability  was  selected  as  a  replacement  term  for  currency  in  an  attempt 
to  further  distinguish  the  concepts.  Perishability  pertains  to  the  fleeting  nature  of  a 
particular  information  item.  For  example,  location  data  for  an  airfield  is  not  as 
perishable  as  the  location  of  a  moving  column  of  tanks. 

In  preparation  for  the  final  site  visit,  the  list  of  utility  factors  was  modified 
by  eliminating  redundant  factors.  The  list  was  expanded  to  include  factors  that 
were  apparently  intended  but  not  explicated  during  previous  interviews  and 
redundant  factors  were  eliminated.  Further,  definitions  were  developed  for  all 
factors.  The  final  site  visit  focused  on  review  of  the  factors  and  their  definitions. 


40 


The  results  of  the  site  visit  indicated  that  all  utility  factors  were  viewed  as  legitimate. 
The  definitions  of  the  factors  as  modified  by  the  site  visit  appear  in  Appendix  E. 

The  list  of  utility  factors  that  operations  users  associate  with  intelligence 
outputs  is  rather  long.  One  of  the  more  interesting  features  of  the  list  is  that  it 
provides  a  broad  view  of  attributes  that  relate  to  utility.  While  numerous  factors 
relate  to  the  content  of  an  output,  other  factors,  such  as  clear  and  concise,  relate  to 
the  way  content  is  presented  in  an  output  rather  than  the  content  itself.  This 
suggests  that  the  user's  perspective  of  utility  includes  not  only  the  actual  content  of 
the  output  but  the  manner  in  which  the  content  is  presented. 

Some  utility  factors  identified  by  users  appear  to  pertain  to  the  intelligence 
process  itself.  The  utility  factor  "filtered,”  for  example,  relates  to  the  utility  users 
place  on  intelligence  being  screened  for  them,  rather  than  having  all  information 
passed  to  them.  A  recurrent  problem  noted  by  some  interview  participants  was 
information  overload,  possibly  the  result  of  insufficient  filtering  on  the  pan  of  the 
intelligence  process.  The  utility  factor  "processed"  relates  to  interpreting  or 
aggregating  information  from  input  level  to  an  output  form.  Again,  this  factor 
appears  to  relate  to  the  intelligence  process  itself  rather  than  the  content  or  style  of 
an  output 

In  summary,  the  interviews  conducted  with  intelligence  users  identified 
numerous  factors  associated  with  the  utility  of  intelligence  outputs.  Some  of  the 
factors  identified  relate  to  attributes  of  the  content  of  an  intelligence  output  Other 
factors  relate  to  the  manner  in  which  the  output  is  presented.  Still  other  factors  in 
user  utility  pertain  to  the  intelligence  process  itself  and  imply  a  quality  control 
function  in  the  process  that  functions  to  increase  value  of  outputs  to  users. 
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Twelve  output  utility  factors  were  identified  by  the  in-house  interview 
participants.  These  utility  factors  were  rated  on  an  importance  scale  for  the  two 
different  contexts,  peacetime  and  active  hostilities.  With  the  exception  of  the  factor 
completeness,  all  factors  were  rated  higher  in  the  active  hostilities  context  than  in 
the  peacetime  context  Figure  7  contains  a  histogram  comparing  the  frequency  of 
importance  ratings  for  active  hostilities  and  peacetime.  The  means  and  standard 
deviations  of  the  importance  ratings  may  be  found  in  Appendix  D.  A  sign  test  was 
conducted  on  the  importance  ratings  of  outputs  and  was  found  to  be  significant 
(z  =  3.67,  p<.0001). 

Accuracy  was  the  highest  rated  utility  factor  in  the  peacetime  context  In  the 
context  of  active  hostilities;  however,  timeliness  and  currency  received  higher 
importance  ratings.  Not  surprisingly,  the  implication  being  that  a  higher  premium 
is  placed  on  receiving  timely  information  during  hostilities.  The  interviews 
confirmed  comments  that  when  there  is  a  trade-off  between  waiting  for  more 
accurate  information  versus  receiving  less  accurate  information  sooner,  users  will 
typically  reduce  their  accuracy  standards  in  favor  of  timeliness  in  an  active 
hostilities  environment 

The  utility  factor  "filtered,"  which  was  discussed  above,  relates  to  multiple 
outputs.  This  factor  was  not  identified  as  a  utility  factor  at  all  sites,  nor  was  the 
factor  identified  by  all  staff  elements.  Unit  operational  environment  and  unit 
mission  may  be  a  possible  influence  on  this  factor.  For  example,  the  information 
overload  problem,  and  hence  the  high  utility  placed  on  filtered  output,  was 
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Figure  7.  Importance  rating  of  output  attributes 
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identified  by  Fort  Stewart  and  Fort  Lewis  operations  personnel  but  not  by  Fort 
Bragg  personnel.  By  contrast,  Fort  Bragg  identified  information  underload  as  a 
key  problem  area.  These  sites  are  contrasted  not  only  by  the  perceived  difference  in 
problems  but  the  distinct  differences  in  mission  orientation. 

Results  of  MAUA  Session 

A  one-day  conference  was  held  at  the  ARI  Fort  Leavenworth  field  office 
with  four  senior-level,  active-duty  Army  personnel  representing  operations  (G3), 
logistics  (G4),  fire  support  control  (FSC),  and  intelligence  (G2).  The  goals  of  the 
conference  were  to  obtain  (a)  consensus  on  a  hierarchy  of  information  items  so  that 
it  represented  a  framework  for  organizing  information  needs;  (b)  consensus  on  a  set 
of  situational  factors  for  defining  situational  context;  and  (c)  importance  weights 
testing  the  hypothesis  that  the  relative  value  of  different  types  of  information 
depends  on  different  users  under  varying  situational  contexts. 

To  prepare  for  the  conference,  the  Army  participants  were  sent  a  package  of 
materials  (see  Appendix  B)  that  included  an  overview  of  the  purpose  of  the 
conference,  a  strawman  hierarchy  of  information  needs,  and  a  description  of  the 
situational  factors  that  the  project  team  hypothesized  were  important  to  measuring 
the  utility  (or  relevance)  of  information  contained  in  MI  outputs.  The  morning 
session  of  the  conference  was  spent  modifying  the  strawman  hierarchy  of 
information  needs  so  that  it  represented  a  framework  that  all  Army  participants 
could  agree  on.  In  addition,  the  participants  independently  generated  "generic 
importance  weights"  indicating,  in  general,  the  relative  value  of  the  information  to 


them  as  specific  types  of  users  of  MI  output.  Unfortunately,  the  logistician  could 
only  participate  in  the  morning  session. 

The  afternoon  session  of  the  conference  was  spent  modifying  the 
description  of  the  situational  factors,  in  particular  the  External  Factors  Matrix,  so 
that  it  represented  those  factors  that  the  participants  agreed  would  affect  the  utility  of 
different  information  needs.  The  three  participants  independently  generated  a 
second  set  of  importance  weights,  bur  this  time  for  a  situation  (defined  in  terms  of 
one  level  from  each  of  the  external  factors)  that  they  thought  was  substantially 
different  from  the  situation  they  had  in  mind  when  generating  the  "generic 
weights."  The  conference  concluded  with  the  participants  completing  the  MI 
Product  Usage  Questionnaire,  and  with  a  general  discussion  of  what  other  factors 
affect  the  utility  of  MI  outputs. 

Although  different  members  of  the  project  team  led  different  sessions  of  the 
conference,  the  format  throughout  the  conference  was  that  of  a  working  session. 
This  section  of  the  report  focuses  only  on  the  obtained  hierarchy  of  information 
needs,  the  external  factors  matrix,  and  the  two  sets  of  importance  weights. 

Information  Needs  Hierarchy 

The  agreed  upon  hierarchy  of  information  needs  is  presented  in  Figure  8. 
The  hierarchy  has  five  top-level  nodes:  I,  Reference  to  Relevance  of  Information; 

II,  Description  of  Battlefield  Area;  III,  Description  of  Enemy  Situation; 

IV,  Description  of  Enemy  COAs;  and  V,  Intelligence  Considerations.  Each  node  is 
subdivided  and  then  in  most  cases  are  further  subdivided  in  order  to  identify  more 
distinct  information  needs. 
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Comparison  with  the  strawman  hierarchy  in  Appendix  B  indicates  that  the 
participants  significantly  modified  the  hierarchy  in  two  ways.  First,  they  made  the 
strawman  hierarchy,  which  was  primarily  built  using  the  categories  found  in  the 
intelligence  estimate,  less  doctrinal  and  more  cognitive  in  the  sense  that  the 
relationship  between  information  nodes  represented  better  what  they  saw  to  be  the 
information  needs.  Second,  where  appropriate,  they  modified  the  structure  itself  to 
correspond  better  with  the  principal  steps  in  the  IPB  process.  In  particular,  they 
modified  the  elements  comprising  "Description  of  Enemy  Situation"  and  combined 
two  nodes  in  the  strawman  hierarchy  to  form  the  "Description  of  the  Enemy 
Courses  of  Action"  for  the  hierarchy  to  better  correspond  with  the  Threat 
Description  and  Threat  Integration  steps,  respectively,  in  the  IPB  process. 

After  reaching  consensus  on  the  hierarchy  of  information  needs,  the 
participants  independently  generated  "generic"  importance  weights  indicating  in 
general  the  relative  importance  they  placed  on  the  items  of  information.  Weighting 
proceeded  in  a  "bottom-up"  manner,  beginning  with  the  weighting  of  the  lowest 
levels  of  the  hierarchy  first  The  weights  were  obtained  by  asking  the  participants 
to  divide  up  100  points  among  the  information  items  at  each  node  of  the  hierarchy. 
For  example,  they  divided  100  points  among  the  three  information  items 
comprising  Weather,  among  the  three  nodes  (including  Weather)  comprising  the 
Description  of  the  Battlefield  Area  node,  and  among  the  five  nodes  which,  in  total, 
comprised  Information  Utility  as  represented  in  the  entire  hierarchy.  Empirical 
research  by  Adelman,  Sdcha,  and  Donnell  (1986)  has  shown  that  this  simple 
technique  of  dividing  up  100  points  is  a  reasonably  good  technique  for  obtaining 
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relative  importance  weights,  and  certainly  adequate  for  the  purpose  of  the 
conference. 

The  "generic"  relative  importance  weights  for  each  of  the  four  participants 
are  shown  in  Table  5.  Before  examining  the  weights  the  reader  should  keep  the 
following  points  (or  caveats)  in  mind.  First,  the  weights  represent  tire  opinion  of 
four  individuals  and  should  not  be  generalized  to  be  necessarily  representative  of 
the  four  different  user  groups.  Second,  the  relative  weights  were  not  discussed 
during  the  conference  and,  consequently,  the  participants  did  not  have  the  benefit  of 
discussion  and  subsequent  clarification  that  might  have  prompted  them  to  modify 
their  position.  Third,  because  of  time  limitations,  the  group  facilitator  did  not  have 
the  opportunity  to  implement  procedures  for  helping  the  participants  scrutinize  their 
position. 

All  the  caveats  notwithstanding,  examination  of  Table  5  indicates  that,  as 
hypothesized,  users  did  significantly  differ  in  the  generic  relative  weights  they  gave 
to  the  hierarchy,  thereby  reflecting  their  different  relative  information  needs.  Figure 
9  offers  a  graphic  representation  of  the  relative  weights  assigned  to  the  five 
high-level  nodes  of  the  hierarchy  for  the  generic  context  Examination  of  the 
relative  weights  for  the  G3  and  the  G4  illustrate  this  point.  The  G3  considered 
"Description  of  Enemy  Courses  of  Action"  to  be  the  most  important  and  gave  it  a 
relative  weight  of  SO.  The  G4  also  considered  "Description  of  Enemy  Courses  of 
Action"  to  be  important  (rel.  wt  »=  25),  but  considered  the  most  important  to  be 
"Description  of  Battlefield  Area  (rel.  wl  ■  45).  Examination  of  the  relative  weights 
given  to  the  two  categories  comprising  the  "Description  of  Enemy  COAs"  also 
demonstrates  the  differing  relative  information  needs  of  the  G3  and  the  G4.  In 
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TABLE  5.  Relative  Importance  Weights:  Wartime 


INFORMATION  ITEM 


I.  REFERENCE  TO  RELEVANCE 


A.  Refen  to  Mission 


B.  Refen  to  CMDR  Guidance 


D.  DESCRIPTION  OF  BATTLEFIELD  AREA 


A.  Weather 


1.  Describes  Situation 

2.  Describes  effects  on  Enemy  Operations 

3.  Describes  effects  on  Friendly  Operations 


B.  Terrain 


1.  Describes  Situation 

2.  Describes  effects  on  Enemy  Operations 

3.  Describes  effects  on  Friendly  Operations 


C.  Other  Characteristics 


1.  Existing  Situation 

2.  Describes  effects  on  Enemy  Operations 

3.  Describes  effects  on  Friendly  Operations 


m.  DESCRIPTION  OF  ENEMY  SITUATION 


A.  Enemy  Disposition  and  Composition 


1.  Forward  Trace 

2.  Unit  Locations 

3.  Main  Efforts 

4.  Combat  Support 

5.  Echelonment 

6.  Reserves 

7.  Staging  Areas 

8.  Combat  Service  Support 

9.  Air  Forces 

10.  C2 


B.  Strength  of  Enemy  Forces  by  Echelon 


1.  Readiness 

Z  Identified  Resupply  Rates 

3.  Identified  Critical  Nodes 

4.  Levels  of  Morale 

3.  Strength  of  Air  Forces 
6.  Nuclear,  Biological,  and  Chemical 


C.  Recent  and  Present  Significant  Activities 


1.  Combat  Action 

2.  Manuever/Movement 

3.  C2 

4.  Sustainment 
3.  Intelligence 
6.  Other 
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33  <40  10 

33  40  70 


20  20  10 


33  60  33 

33  10  33 

33  30  33 


20  15  10 


40  50  30 


15  20 

20  0 
20  20 


EiM 

1 


12  |  10 
5 


40  30  20 


40  |  55  10 

5  10 

15  5 

10  5 

15  5 

15  20  I  30 


20  20  50 


25  70 

25  ' 

20  10 

10  10  10 

20  20  20 

10  10  0 
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TABLE  5.  Relative  Importance  Weights:  Wartime  Continued. 
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Figure  9.  Comparison  of  user's  relative  importance  weights  in  wartime. 
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particular,  the  G3  gave  "Enumerates  Possible  COAs"  a  relative  weight  of  10  and 
"Other  Information  for  Probable  Enemy  Courses  of  Action"  a  relative  weight  of  90. 
In  contrast,  the  G4  gave  all  100  points  to  "Enumerates  Possible  Courses  of 
Action."  Examination  of  the  ten  items  comprising  "Enumerates  Possible  Course  of 
Action"  again  shows  that  the  G3  and  G4  had  a  different  distribution  of  relative 
weights,  again  reflecting  differing  relative  information  needs. 

When  examining  Table  5  the  reader  should  keep  two  additional  points  in 
mind.  First,  the  G3  and  the  G4  did  not  always  have  differing  relative  information 
needs.  And,  second,  while  the  G3  and  the  G4  appear  to  have  the  greatest 
difference  in  their  relative  information  needs,  the  G2  and  F SC  also  had  differences 
in  their  relative  information  needs,  both  between  themselves  and  with  the  G3  and 
G4.  Examination  of  the  relative  weights  the  participants  gave  the  three  items 
comprising  the  "Other  Characteristics"  attribute  category  under  the  "Description  of 
Battlefield  Area"  node  illustrates  both  points.  Specifically,  the  G3  and  the  G4  gave 
all  three  attributes  equal  weight.  In  contrast,  the  G2  gave  the  higher  relative  weight 
and  the  FSC  the  lowest  relative  weight  to  "existing  situation." 

External  facias 

The  second  goal  of  the  conference  was  to  obtain  consensus  on  a  set  of 
external  factors  far  defining  situational  context  During  the  course  of  the 
conference  the  external  factors  matrix  was  streamlined.  Participants  in  the 
conference  pointed  out  that  the  distinctions  between  elements  in  all  the  categories 
were  so  subtle  that  utility  weighting  lost  its  impact  For  example,  in  Levels  of 
Conflict  from  Terrorism  to  High  Intensity,  the  nuances  were  impossible  to  account 
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for  discretely.  However,  in  a  case  where  there  are  only  three  variations  on  the 
situational  theme,  such  as  Peace,  Resources  Short  of  War,  and  War,  there  was  a 
suitably  powerful  distinction.  (Resources  Short  of  War  is  a  new  term  being  applied 
to  several  levels  of  conflict  from  terrorism  through  low  intensity  conflict)  Similar 
modifications  were  made  under  each  situational  component  Equally  as  important, 
conference  participants  pointed  out  that  the  order  in  which  the  column  headings 
were  arrayed  should  be  modified.  Their  recommendation  was  that  from  left  to 
right,  the  first  elements  must  be  echelon  and  level  of  conflict  because  all  other 
external  factors  relate  in  logical  sequence  from  those  initial  two. 

Other  recommended  changes  included:  (1)  use  of  the  SIGMA  STAR 
functions  rather  than  identifying  user  staff  positions,  (2)  aggregation  of  echelons 
into  three  relevant  elements  that  share  common  attributes,  (3)  reduction  of  the 
mission  possibilities  to  three  primary  elements,  and  (4)  inclusion  of  Training  in  the 
Operational  Applications  field.  The  result  of  the  modifications  is  a  more  usable 
matrix,  shown  in  Table  6,  from  which  situational  variables  may  be  selected.  The 
selected  variables  yield  a  contextual  profile  that  does  permit  a  weighted  evaluation 
of  information  utility  in  a  situational  context 

If  one  assumes  that  the  project  team  will  focus  only  on  the  echelon  level  of 
"division  and  below,"  and  that  relative  weights  have  to  be  obtained  for  each  of  the 
five  user  groups,  then  there  are  36  (3x3x4)  situations,  defined  in  terms  of  the  other 
possible  combinations  of  the  levels  on  the  other  possible  combinations  of  the  levels 
on  the  other  external  factors,  that  might  significantly  affect  the  relative  importance 
of  users’  information  needs.  Discussion  with  conference  participants  suggests, 
however,  that  it  might  be  possible  to  group  the  36  situations  into  perhaps  no  more 
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TABLE  6.  Revised  External  Factors  Matrix. 


ECHELON 

LEVEL  OF  CONFLICT 

USER 

MISSION 

APPLICATION 

National 

Peacetime/Noncombat 

Maneuver 

Garrison 

1 

Data  Base/ 

Record  Keeping 

Corps/ 

Theater 

Resources  Short  of  War 

Fires 

Deployment 

Planning 

Division 
and  below 

War 

Combat 

Service 

Support 

Combat 

Operations 

Execution 

Air  Defense 

Training 

Intelligence 
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than  five  clusters  such  that  each  user's  relative  weights  for  the  information  needs 
would  be  the  same  for  situations  within  a  cluster,  but  different  for  each  cluster.  The 
possibility  of  clustering  situations  needs  to  be  pursued  because  it  would  minimize 
the  extent  to  which  the  utility  of  MI  outputs  must  be  treated  as  situadonally 
dependent  and,  in  turn,  would  simplify  the  development  and  application  of  a  utility 
metric  for  MI  outputs. 

Situational  Information  Needs 

After  obtaining  consensus  on  the  External  Factors  Matrix,  the  participants 
were  asked  if  they  were  thinking  of  a  situation  defined  in  terms  of  one  level  of  each 
of  the  four  external  factors  except  "user"  when  they  generated  their  generic  relative 
weights  for  the  information  items.  All  participants  indicated  that  they  were  basically 
thinking  of  the  following  situation:  division  and  below,  war,  combat  operations, 
and  execution.  The  participants  were  then  asked  if  they  could  define  a  situation  for 
which  they  would  have  different  weights  for  the  information  items.  They 
suggested  the  situation  of  "division  and  below,  peace/noncombat,  garrison,  and 
data  base/record  keeping."  The  relative  weights  for  both  situations,  respectively, 
for  the  G2,  G3,  and  FSC  are  presented  in  Table  7. 

Examination  of  the  relative  weights  that  each  participant  gave  for  both 
situations  clearly  supports  the  hypothesis  that  the  relative  importance  of  the 
information  needs  is  situadonally  dependent  For  example,  "Description  of  the 
Battlefield  Area"  and  "Description  of  the  Enemy  Situation"  were  the  two  most 
important  information  needs  for  the  G2  for  the  second  situation  and  the  two  least 
important  needs  for  the  first  situation.  The  "Reference  to  Relevance"  was  the  most 


TABLE  7.  Comparison  of  Relative  Importance  Weights. 


Peacetime 

Wartime 

INFORMATION  ITEM 

G3 

G2 

FSC 

G3 

G2 

FSC 

I.  REFERENCE  TO  RELEVANCE 

50 

10 

30 

10 

25 

20 

A.  Refen  to  Mission 

70 

60 

40 

50 

30 

50 

B.  Refers  to  CMDR  Guidance 

30 

40 

60 

50 

70 

50 

IL  DESCRIPTION  OF  BATTLEFIELD  AREA 

10 

30 

10 

10 

IS 

10 

A.  Weather 

30 

15 

30 

40 

30 

40 

1.  Describes  Situation 

40 

60 

25 

33 

40 

B 

2.  Describes  effects  on  Enemy  Operations 

20 

IS 

33 

20 

19 

3.  Describes  effects  on  Friendly  Operations 

40 

20 

60 

33 

50 

B.  Terrain 

30 

70 

50 

40 

50 

40 

1.  Describes  Situation 

mm 

20 

Q 

33 

B 

33 

2.  Describes  effects  on  Enemy  Operations 

K9 

40 

n 

33 

Hi 

33 

3.  Describes  effects  on  Friendly  Operations 

HI 

40 

60 

33 

40 

33 

C.  Other  Characteristics 

40 

15 

20 

20 

20 

20 

1.  Existing  Situation 

20 

60 

33 

60 

B 

2.  Describes  effects  on  Enemy  Operations 

10 

40 

10 

33 

10 

19 

3.  Describes  effects  on  Friendly  Operations 

40 

40 

30 

33 

30 

40 

m.  DESCRIPTION  OF  ENEMY  SITUATION 

10 

30 

10 

20 

15 

20 

A.  Enemy  Disposition  and  Composition 

20 

55 

20 

40 

50 

40 

1.  Forward  Trace 

0 

0 

5 

10 

15 

10 

2.  Unit  Locations 

10 

25 

10 

10 

20 

15 

3.  Main  Efforts 

n 

0 

10 

15 

20 

15 

4.  Combat  Support 

n 

5 

10 

8 

5 

5 

S.  Echelonment 

Efl 

15 

5 

8 

10 

5 

6.  Reserves 

10 

15 

10 

12 

10 

10 

7.  Staging  Areas 

10 

10 

8 

5 

15 

8.  Combat  Service  Support 

10 

5 

10 

12 

5 

5 

9.  Air  Forces 

10 

10 

10 

B 

5 

5 

10.  C2 

10 

15 

20 

HI 

5 

15 

B.  Strength  of  Enemy  Forces  by  Echelon 

20 

35 

20 

40 

30 

20 

1.  Readiness 

30 

25 

30 

40 

55 

25 

2.  Identified  Resupply  Rates 

0 

10 

10 

5 

5 

6 

3.  Identified  Critical  Nodes 

5 

25 

15 

15 

5 

25 

4.  Levels  of  Morale 

5 

5 

15 

10 

5 

6 

S.  Strength  of  Air  Force* 

30 

20 

10 

15 

5 

13 

6.  Nuclear,  Biological,  and  Chemical 

30 

15 

20 

15 

20 

25 

C  Recent  md  Present  Significant  Activities 

60 

o 

60 

20 

20 

40 

0 

0 

mm 

wm 

B 

2.  Msnueve/M overrent 

5 

10 

H 

19 

25 

E9 

3.  C2 

20 

30 

30 

20 

10 

20 

4.  Sustainment 

20 

25 

15 

10 

10 

15 

3.  Intelligence 

50 

25 

10 

20 

20 

10 

6.  Other 

5 

10 

5 

10 

10 

5 
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TABLE  7.  Comparison  of  Relative  Importance  Weights  Continued. 


Peacetime 

Wartime 

INFORMATION  ITEM 

G3 

G2 

FSC 

G3 

G2 

FSC 

10 

R 

20 

50 

25 

30 

A.  Enumerate*  Possible  ECOAs 

30 

60 

70 

10 

70 

60 

1.  Mission 

20 

0 

20 

4 

15 

2.  Objective 

20 

10 

10 

4 

10 

10 

3.  Forces 

20 

25 

10 

9 

15 

10 

4.  Terrain  Considerations 

0 

20 

d 

2 

mm 

5.  Echelonment 

10 

20 

9 

6.  Main/Supporting  Effort 

Kfl 

5 

10 

10 

IB 

7.  Fires  (includes  Air  Support) 

Lit 

10 

15 

10 

8.  Time/Distance  Factors 

H 

WE 

5 

14 

5 

o 

9.  Threat  Advance 

10 

10 

0 

14 

10 

1 

10.  Probability 

20 

D 

20 

23 

15 

10 

B .  Other  Information  for  Probable  ECOAs 

70 

40 

30 

90 

30 

40 

1.  Center  of  Gravity 

B9 

15 

El 

60 

25 

2.  Vulnerabilities 

1 

50 

60 

25 

3.  High  Value  Targets 

i 

35 

□ 

10 

30 

50 

V.  INTELLIGENCE  CONSIDERATIONS 

20 

10 

30 

10 

20 

20 

A.  Critical  Uncertainties 

80 

40 

40 

70 

31 

30 

B.  PIRs 

20 

30 

15 

20 

mm 

warn 

C.  Information  Requirements 

jj^^H 

10 

15 

19 

10 

D.  NAls 

SI 

20 

30 

10 

m 

■■ 
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imponant  criterion  to  the  G3  in  the  second  situation  and  the  least  important  in  the 
first  situation.  In  contrast,  "Description  of  Enemy  Courses  of  Action"  was  the  most 
important  of  information  needs  to  the  G3  in  the  first  situation  and  tied  for  least 
important  in  the  second  situation.  The  "Reference  to  Relevance"  and  "Intelligence 
Considerations"  increased  somewhat  in  importance  and  the  "Description  of  Enemy 
Courses  of  Action"  decreased  somewhat  in  importance  in  the  second  situation  for 
the  FSC.  However,  in  comparison  to  the  G2  and  G3,  the  FSC's  relative  weights 
were  not  as  situationally  dependent. 

Figure  10  offers  a  comparison  of  the  relative  weights  of  the  G3  and  G2  for 
die  contexts  of  wartime  and  peacetime.  From  this  comparison  it  is  obvious  that  the 
G3  and  G2  have  different  priorities  for  information.  It  is  interesting  to  note  the 
fairly  wide  variation  in  weighting  in  one  context  versus  the  other.  In  terms  of  the 
rank  ordering  of  relative  weights,  the  comparison  of  peacetime  to  wartime 
demonstrates  a  reordering  of  priorities  of  the  high-level  factors  for  both  the  G3  and 
G2.  This  offers  evidence  for  the  need  to  elicit  weights  within  explicit  contexts 
defined  by  the  external  factors. 

MAUA  Conference  Summary 

Although  further  research  is  obviously  required,  the  results  obtained  during 
the  Fort  Leavenworth  conference  are  quite  encouraging.  In  total,  they  demonstrated 
that  MAUA  represents  a  viable  approach  for  (1)  better  understanding  MI  users' 
subjective,  concept  of  utility,  and  (2)  providing  an  explicit,  quantitative  approach 
for  measuring  the  utility  of  different  items  of  information  for  various  users  under 
different  situational  contexts.  These  were  the  two  principal  goals  of  our  efforts  to 
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Figure  10.  Comparison  of  relative  importance  weights  of  G3  and  G2. 


apply  a  MAUA  approach  within  the  first  year  of  the  project  Moreover,  it  is  the 
hypothesis  of  members  of  the  project  team  that  MAUA  can  also  be  used  to  measure 
other  utility  dimensions  inherent  in  MI  output  as  well.  The  multidimensional  nature 
of  utility  will  certainly  be  a  major  focus  of  research  in  Year  2  of  the  project 
The  most  surprising  and  possibly  most  important  result  of  the  MAUA 
conference  relates  to  the  hierarchy  of  information  items.  The  strawman  hierarchy 
developed  prior  to  the  conference  was  doctrinal-based,  drawn  primarily  from  the 
intelligence  estimate.  Conference  participants  offered  significant  modifications  to 
the  hierarchy  in  an  effort  to  fit  their  needs  from  intelligence  outputs.  This  modified 
hierarchy  is  non-doctrinal  in  nature,  though  the  Fort  Leavenworth  participants 
described  the  reorientation  as  "emerging  doctrine"  rather  than  non-doctrinal.  In 
contrast  to  the  hierarchy  related  to  the  intelligence  estimate,  the  modified  hierarchy 
relates  to  principal  steps  in  the  IPB  process.  The  relationship  between  the 
intelligence  process  and  intelligence  users  will  be  facilitated  greatly  by  the  IPB 
linkage  to  intelligence  outputs. 

Results  of  Military  Intelligence  Product  Usage  Questionnaire 
The  MI  PUQ  was  distributed  to  current  Army  officers  who  participated  in 
the  individual  interviews  and  the  decision  conferences  held  at  Fort  Leavenworth  and 
Fort  Carson.  A  limited  number  of  questionnaires  was  sent  to  interested  current 
Army  officers  that  could  not  be  visited  for  the  individual  interviews.  The  data 
reported  here  were  collected  during  the  first  year  and  well  into  die  second  year  of 
this  research.  The  findings  reported  here  are  based  on  the  questionnaire  responses 
of  23  individuals,  who  are  currently  serving  in  active  duty  positions. 
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The  ranks  of  individuals  completing  the  questionnaire  included:  Lieutenant 
Colonel  (10);  Major  (1 1);  Captain  (2).  19  of  the  individuals  participating  in  the 
survey  are  currently  serving  in  division  staffs.  The  remaining  4  included  in  the 
survey  are  serving  as  instructors  at  the  Command  and  General  Staff  College. 

The  individuals  who  participated  in  this  survey  included:  1 1  individuals 
from  operations  (including  G3s,  G3  Plans,  and  G3  Operations;  2  from  intelligence 
(both  G2s);  5  from  fire  support  (FS COORD);  3  chemical  officers;  and  2  from 
logistics  (including  one  G4).  Given  the  large  number  of  operations  personnel  in  the 
survey,  the  data  are  reported  by  group:  the  operations  group  (N  =  11)  and  the  non- 
operations  group  (N  =  12). 

jntsiiige,nce.Pp^y9t£amiIiamy 

Detailed  descriptive  statistics  for  product  familiarity,  including  mean, 
standard  deviation,  median,  and  maximum  and  minimum  ratings,  are  contained  in 
Appendix  F.  Table  8  contains  the  mean  familiarity  response  for  each  product  and 
groups  of  products  by  user  groups.  The  mean  figure  for  the  groups  of  products 
(e.g.,  doctrinal  products)  was  derived  from  individual  participant's  mean  rating  of 
each  product  group. 

A  considerable  range  of  familiarity  with  the  intelligence  products  is  evident 
from  a  review  of  Table  8.  For  the  operations  users,  briefings  received  the  highest 
familiarity  rating.  For  the  non-operations  users,  spot  reports  and  situation  reports 
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TABLE  8.  Intelligence  Product  Familiarity, 


PRODUCT 


Doctrinal  Products: 


INTSUM 


PERINTREP 


SUPINTREP 


Intelligence  Annex 


Intelligence  Estimate 


INTREP 


SITREP 


Imagery  Reports 


Spot  Report 


Responses  to  Information 
Requests 


Briefings: 


Formal 


Informal 


Non-Doctrinal  Products: 


Collection  Priorities 


Target  List 


Target  Nomination 


Target  Priority  List 


MEAN  RATING 

OPERATIONS  USERS  INON-OPERATIONS  USERS 


3.8 


NAIs 

4.1 

TAIs 

4.1 

FAMILIARITY  SCALE 

5  -  Extremely  Familiar  3  -  Moderately  Familiar  1  -  Not  at  All 

Ftmiliar 

4  -  Very  Familiar  2  -  Slightly  Familiar 


62 


received  the  highest  familiarity  rating.  Imagery  reports  received  the  lowest 
familiarity  rating  from  both  groups  of  users. 

The  mean  familiarity  rating  for  the  doctrinal  products  as  a  group  is  quite 
close  for  both  user  groups.  The  largest  difference  in  familiarity  is  found  with  the 
non-doc trinal  products,  the  group  of  operations  users  showing  a  higher  mean 
familiarity  than  the  non-operations  group.  While  a  difference  in  familiarity  ratings 
exists  between  the  groups,  the  inclusion  of  non-doctrinal  products  in  the  list  of 
intelligence  products  did  not  appear  to  be  problematic  in  terms  of  recognition  of 
these  items  as  products  by  the  majority  of  respondents. 

Intelligence  Product  Usefulness 

Product  usefulness  ratings  were  compiled  for  all  products  receiving  a 
minimum  familiarity  rating  of  2.  In  this  manner,  usefulness  data  were  not  accepted 
from  survey  participants  who  indicated  that  they  were  "Not  at  All  Familiar"  with  a 
particular  product  The  reason  for  this  minimum  familiarity  requirement  was  based 
on  the  need  for  compiling  opinions  about  products  that  had  a  basis  in  the 
respondent's  experience  rather  than  doctrine  or  some  unidentifiable  source. 

Detailed  descriptive  statistics  for  product  usefulness,  including  mean, 
standard  deviation,  median,  and  maximum  and  minimum  ratings,  are  contained  in 
Appendix  F.  The  mean  usefulness  ratings  by  user  group  for  all  products  and 
products  groups  are  contained  in  Table  9. 

While  the  mean  usefulness  ratings  for  the  group  of  doctrinal  products  is 
rather  close  between  the  two  user  groups,  a  sizable  difference  was  shown  in  the 
ratings  of  the  non-doctrinal  group  of  products  (which  was  significant  on  the 
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TABLE  9.  Intelligence  Product  Usefulness. 


PRODUCT 


Doctrinal  Products 


INTSUM 


PERINTREP 


SUPINTREP 


Intelligence  Annex 


Intelligence  Estimate 


INTREP 


SITREP 


Imagery  Reports 


Spot  Report 


Responses  to  Information 
Requests 


Briefings: 


Formal 


Informal 


Non-Doctrinal  Products 


Collection  Priorities 


NAls 


TAIs 


Target  List 


Target  Nomination 


Target  Priority  List 


MEAN  RATING 

(OPERATIONS  USERS  I  NON-OPERATIONS  USERS 


USEFULNESS  SCALE 

5  •  Extremely  Useful 

3  -  Moderately  Useful  1  -  Not  Useful 

4  -  Very  Useful 

2  •  Slightly  Useful 
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Wilcoxon  Signed  Rank  test).  The  products  contained  in  the  non-doc trinal  product 
group  include  numerous  outputs  of  the  IPB  process.  The  high  usefulness  ratings 
of  the  non-doctrinal  products  by  the  operations  group  may  indicate  that  IPB  type 
outputs  are  of  prime  utility  to  operations  users. 

Examination  of  Table  9  offers  an  interesting  comparison  of  product 
usefulness  for  the  two  groups  of  users  surveyed.  For  the  operations  group  of 
users,  six  products  received  a  mean  rating  of  4.0  or  higher,  compared  to  four 
products  for  the  non-operations  group.  Two  products,  the  intelligence  estimate  and 
the  INTSUM,  received  high  ratings  from  both  groups.  While  two  non-doctrinal 
products,  TAIs  and  NAIs,  were  in  the  top  six  products  for  the  operations  group, 
they  were  among  the  lower  rated  products  of  the  non-operations  group.  Both  user 
groups  rated  informal  briefings  higher  than  formal  briefings.  Within  the  group  of 
doctrinal  products  both  groups  gave  rather  low  ratings  to  the  same  products, 
SUPINTREPs,  imagery  reports,  and  responses  to  information  requests. 

Jmeiiigcngg.toiMCt  &gqwncY  of  Use 

As  noted  earlier,  the  rating  of  frequency  of  product  use  was  not  restricted  to 
a  single  response  per  product  The  data  were  summarized  in  the  following  manner 
for  each  group  of  products  (e.g.,  doctrinal  products),  a  count  was  made  of  the 
number  of  responses  per  category  (e.g.,  "Don't  Use");  this  was  then  converted  to  a 
percentage  of  category  responses  to  the  total  number  of  responses  on  that  product. 
Appendix  F  contains  the  response  percentage  to  the  question  for  each  product  by 
user  group.  The  data  reported  on  frequency  of  product  use  are  restricted  to  those 
products  receiving  a  minimum  familiarity  rating  of  2. 
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Table  10  offers  a  summary  of  the  percentage  of  responses  of  each  category 
of  users  for  each  product  group.  The  response  percentage  for  frequency  of  use 
categories  for  the  group  of  doctrinal  products  is  quite  similar  for  both  operations 
and  non-operations  users.  Doctrinal  products  received  a  slightly  higher  percentage 
of  "Don't  Use"  responses  from  non-operations  users.  Considerable  differences 
existed  between  the  groups  on  their  frequency  of  use  responses  regarding  briefings. 
Regarding  non-doctrinal  products,  the  two  groups  of  respondents  differed  to  a 
considerable  extent  in  two  categories,  namely  "Don't  Use"  which  was  higher  for 
the  non-operations  group,  and  "Use  Repeatedly,"  which  was  considerably  higher 
for  the  operations  group.  Differences  existed  between  the  groups  on  all  categories 
of  frequency  of  use  of  briefings.  The  largest  difference  was  seen  on  the  categories 
of  "Use  Repeatedly"  (higher  for  the  operations  users),  "Use  Once"  (higher  for  the 
non-operations  users),  "Use  Right  Away"  (higher  for  the  operations  users). 


As  discussed  earlier,  this  portion  of  the  questionnaire  was  added  after  the 
first  data  collection  effort  The  number  of  respondents  reflected  in  the  following 
results  discussion  is  based  on  14  individuals,  5  operations  users,  and  9 
non-operations  users. 

Respondents  were  permitted  to  select  as  many  categories  as  needed  to 
classify  their  reason  for  using  a  particular  product  Respondents  were  given  four 
categories  to  select  from  in  addition  to  an  "other"  category  which  they  were  asked  to 
specify.  The  categories  of  "Only  Source"  and  "Best  Source"  appeared  to  be  viewed 
as  mutually  exclusive  to  respondents  in  that  none  of  the  products  were  classified  as 
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TABLE  10.  Intelligence  Product  Frequency  of  Use. 


Product  Type  and  User  Group 

Don't 

Use 

File 

Use 

Once 

Operations  Users 

7% 

7% 

28% 

31% 

1 

26% 

Non-Operations  Users 

12% 

7% 

29% 

29% 

24% 

Briefings 

Operations  Users 

0 

0 

19% 

58% 

23% 

Non-Operations  Users 

4% 

7% 

29% 

29% 

32% 

Non-Doctrinal  Intelligence 
Products 

Operations  Users 

4% 

6% 

21% 

50% 

19% 

Non-Operations  Users 

14% 

11% 

19% 

35% 

21% 
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both  "only"  and  "best.”  A  frequency  of  response  was  calculated  for  the  products  in 
the  same  manner  as  discussed  in  the  results  of  the  frequency  of  use  question. 
Appendix  F  contains  the  response  percentage  for  each  product  As  with  the 
previous  questions,  only  those  products  receiving  a  minimum  familiarity  rating  of  2 
are  included  in  the  reporting. 

Few  intelligence  products  received  the  "Only  Source  of  Information"  reason 
for  use  by  both  user  groups,  namely  SUPINTREPs,  imagery  reports,  and  formal 
briefings.  Those  three  reports  were  the  only  ones  given  the  only  source  reason  for 
use  by  the  operations  user  group.  Wi»v 've  exception  of  the  SITREP,  spot  report, 
informal  briefing,  and  TAIs,  all  intelligence  products  received  only  source  of 
information  responses  from  the  non-operations  user  group. 

The  non-operations  group  responded  with  the  "Best  Source"  reason  for  use 
for  all  products,  though  in  many  cases  the  percentage  of  responses  was  quite  low. 
The  operations  group,  however,  did  not  give  a  "Best  Source"  reason  for  use  to  all 
products.  In  particular,  imagery  reports,  formal  and  informal  briefings,  target  list, 
and  target  nominations  were  not  identified  as  used  because  these  were  the  best 
source.  "Easy  to  Use"  was  the  reason  for  use  of  the  target  list  and  target 
nomination  products,  and  to  a  lesser  extent  the  reason  for  use  of  briefings. 

There  was  little  agreement  between  the  operations  and  non-operations  user 
groups  on  the  "Easy  to  Use"  reason  for  product  use.  The  only  product  both  groups 
agreed  on  was  the  SUPINTREP,  which  received  one  third  of  the  responses  in  each 
group.  The  INTSUM,  for  example,  received  80%  of  the  responses  in  the  "Easy  to 
Use"  category  in  the  operations  group,  while  receiving  only  10%  in  the  non¬ 
operations  group. 
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Both  the  operations  and  non-operations  user  groups  agreed  that  the  primary 
reason  for  use  of  SITREP  and  the  spot  report  was  the  response  "Most  Timely." 

The  groups  also  agreed  on  a  number  of  products  that  are  not  used  because  of  the 
timeliness  aspect  (including  the  INTSUM,  SUPINTREP,  collection  priorities, 
target  list,  target  nomination,  and  target  priority  list). 

Table  1 1  provides  a  summary  of  the  reason  for  use  of  the  product  groups  by 
user  groups.  Review  of  this  table  shows  substantial  differences  between  the  user 
groups.  The  most  frequent  reason  for  use  of  all  products  by  the  operations  user 
group  was  "Easy  to  Use."  The  non-operations  user  group,  on  the  other  hand,  most 
frequently  selected  "Best  Source"  as  the  reason  for  use  of  both  doctrinal  and  non- 
doctrinal  products. 


Suggested  Product  Improvements 

Offering  suggestions  for  product  improvement  was  at  the  discretion  of  the 
survey  respondent  Numerous  participants  took  the  opportunity  to  suggest 
improvements  for  some  intelligence  products.  While  all  recommendations 
suggested  by  survey  participants  are  contained  in  Appendix  F  only  a  few  will  be 
noted  here.  The  recommendation  to  increase  graphics  appears  a  number  of  times 
for  various  products.  One  respondent  to  the  questionnaire  suggested  a  new  format 
for  the  intelligence  summary,  termed  a  graphic  INTSUM.  This  respondent 
indicated  that  his  division  uses  the  graphic  INTSUM.  This  individual,  a  division 
G3,  noted  that  the  graphic  INTSUM  was  extremely  useful,  and  characterized  his 
reasons  for  using  it  as  Best  Source  Available,  Easy  to  Use,  and  Most  Timely.  He 
further  suggested  the  need  for  the  widespread  use  of  the  graphic  INTSUM. 
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TABLE  11.  Intelligence  Product  Reason  for  Use. 


Product  Type  and  User  Group 

Only  Source 

Best  Source 

Easy  to  Use 

Most  Timely 

Operations  Users 

5% 

28% 

37% 

30% 

Non-Operations  Users 

15% 

37% 

'  24% 

25% 

Briefings 

Operations  Users 

20% 

0 

70% 

10% 

Non-Operations  Users 

11% 

11% 

33% 

44% 

Operations  Users 

0 

36% 

48% 

16% 

Non-Operations  Users 

28% 

39% 

33% 

0 
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In  general,  few  products  were  added  to  the  survey  by  respondents.  In 
addition  to  the  graphic  INTSUM  noted  above,  one  participant  from  an  intelligence 
background  added  the  situational  template,  decision  support  template,  and  courses 
of  action  overlay.  This  individual  recommended  the  use  of  decision  support 
graphics  supported  by  a  short  written  matrix  to  convey  intelligence  information. 
Further,  he  noted  the  value  of  templates  as  intelligence  products. 

Open-Ended  Questions 

As  discussed  in  the  technical  approach,  survey  respondents  were  not 
required  to  answer  the  open-ended  questions.  However,  many  respondents  did 
elect  to  answer  the  questions.  Some  survey  respondents  provided  very  lengthy 
discussions  about  their  views  and  experience  with  intelligence,  as  opposed  to  short 
answers  to  each  of  the  questions.  Appendix  F  contains  all  of  the  written  responses 
received  in  the  survey.  Table  12  provides  a  few  selected  comments  from  those 
obtained.  It  should  be  noted  that  not  all  of  the  comments  included  came  from 
individuals  interviewed  during  the  individual  interview  portion  of  the  research. 
While  the  comments  are  not  attributed  to  specific  individuals,  the  comments  are 
those  of  a  range  of  intelligence  users  including:  G3s,  G3Plans,  G2,  FSC,  and  G4. 

A  review  of  the  comments  contained  in  Table  12  highlights  some  common 
concerns  about  intelligence.  The  need  for  intelligence  to  go  beyond  their  data  to 
predictions  is  quite  evident  This  concern  will  be  addressed  in  our  future  research 
concerning  the  evaluation  of  intelligence  output  The  concern  about  information 
overload  was  noted  in  comments,  providing  further  confirmation  to  issues  raised  in 
the  individual  interviews.  One  respondent  in  particular  raised  concerns  about  the 
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TABLE  12.  Selected  Comments  from  Survey  Participants 


"We  must  teach  risk  versus  gain;  we  must  teach  'how  to  think';  the  process  if  properly  taught  will 
result  in  a  more  timely  and  credible  product.  Good  analysts  and  good  analysis  is  not  pontification  but 
a  usable  credible  product,  i.e.  the  (enemy)  is  here  in  this  strength  and  he  is  going  to  do  this." 


"Intelligence  reporting  is  not  rote  and  the  success  of  intelligence  reporting  is  not  measured  by  pages 
produced  but  rather  in  lives  saved  or  lost" 


"(Intelligence)  needs  to  be  more  specific  at  the  risk  of  being  wrong." 


"Intelligence  reports  should  go  farther  in  stating  or  estimating  the  impact  of  new  information  as  it 
relates  to  the  tactical  situation  and  the  capabilities/probable  courses  of  action  available  to  the  enemy. 
That  is  how  does  it  correlate  with  what  we  already  knew." 

'Tve  never  questioned  the  credibility  of  intelligence  analysts  -  they've  always  proven  to  know  a  lot 
more  about  enemy  doctrine,  organization  for  combat  capabilities,  current  situation  than  me. 
Sometimes  I  have  disagreed  with  their  conclusions  -  but  it's  ok  to  be  wrong  as  long  as  an  earnest 
effort  was  made." 


"The  biggest  problem  I'd  had  is  in  getting  the  analyst  to  commit  himself.  They  always  stop  short  of 
saying  this  is  what  I  believe  the  enemy  is  going  to  do." 


"Overall,  reduce  the  bulk,  size,  and  volume  of  (intelligence)  reports." 


"Generally,  the  most  significant  shortcoming  relative  to  intelligence  involves  the  overwhelming 
volume  of  information  their  officers  produce.  Suggest  commander  receive  critical  information  only." 


"Brevity  is  the  key  to  future  needs.  Short,  concise  pieces  of  very  usable  information  that  can  be 
transmitted  via  computers." 
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possibility  of  the  automated  transmission  of  information  in  the  future  and  how  it 
will  require  changes  in  the  way  intelligence  is  communicated. 

Respondents  were  asked  to  specify  any  assumptions  that  they  may  have 
made  about  die  intelligence  products  when  making  their  ratings.  Some  respondents 
indicated  the  assumption  that  the  products  were  delivered  in  a  timely  manner.  One 
respondent  indicated  that  his  appreciation  for  timeliness  was  included  in  his 
judgment  (noting  that  reports  such  as  the  INTSUM  and  PERINTREP  are  always 
delivered  long  after  the  ''cut-ofF'  date  of  the  report  and  spot  reports  are  more 
timely).  Some  respondents  indicated  that  they  assumed  accuracy  in  the  products. 
One  respondent  indicated  that  he  assumed  the  analyst  and  his  products  were 
credible  and  that  the  analyst  was  knowledgeable  of  the  enemy/situation  he  was 
analyzing.  One  respondent  noted  that  the  credibility  of  intelligence  starts  very  low 
and  may  improve  over  time. 

The  final  open-ended  question  in  the  survey  asked  the  respondents,  if  given 
the  purpose  of  our  study  of  the  operational  perspective  of  military  intelligence,  was 
there  anything  they  would  like  to  point  out  to  the  researchers?  Some  individuals 
offered  their  views.  One  respondent  suggested  that  we  look  at  the  physical  location 
of  elements  of  the  intelligence  shop  and  operations  shop  in  garrison  as  well  as  in  a 
tactical  scenario.  While  the  respondent  did  not  elaborate  the  point,  it  might  be  an 
interesting  notion  to  look  at  whether  or  not  physical  proximity  is  a  predictor  of  user 
satisfaction  with  information  provided  or  credibility  of  intelligence  producers  or 
whether  close  proximity  fosters  more  informal  exchanges.  Another  respondent 
suggested  that  intelligence  should  spend  greater  effort  on  the  development  of 
intelligence  and  reduce  the  information  provided  to  the  operational  user. 


Non-doc trinal  products,  specifically  results  of  the  IPB  process,  are  familiar 
to  both  operations  and  non-operations  users.  Further,  in  the  opinion  of  the 
operations  users  participating  in  this  survey,  these  non-doc  trinal  products  are 
considered  more  useful  than  many  doctrinal  products,  though  this  view  is  not 
shared  by  non-operations  users.  Non-doctrinal  products  were  viewed  by  the 
operations  user  group  as  quite  familiar,  quite  useful,  repeatedly  used,  and 
easy  to  use.  These  findings  lead  to  the  conclusion  that  the  IPB 
products  contained  in  the  survey  are  heavily  oriented  toward  the  operations  user, 
certainly  more  so  than  toward  the  non-operations  users. 

Methodology  Results 

A  number  of  lessons  were  learned  from  the  various  methodologies  used  in 
the  research  conducted  to  date.  Open-ended  interviews  tend  to  be  time-consuming 
by  their  nature.  This  is  particularly  the  case  when  participants  must  recall 
information,  such  as  listing  information  needs  for  specific  contexts.  Preparation  of 
handouts  that  participants  can  refer  to  in  identifying  their  requirements  appeared  to 
be  quite  advantageous  in  our  work.  The  prepared  list  served  two  functions:  (1)  to 
cue  participants  of  items  that  they  might  not  remember,  and  (2)  to  save  time. 

As  noted  earlier,  "read-ahead”  packages  were  prepared  and  sent  to 
participants  in  the  Fort  Leavenworth  conference  in  advance  of  the  conference.  The 
read-ahead  package  was  intended  to  provide  each  participant  with  an  understanding 
of  the  objectives  of  the  conference  and  the  research  effort  as  well  as  to  provide 


detailed  strawman  structures  of  information  needs  and  the  situational  context  The 
use  of  read-ahead  packages  proved  to  have  a  number  of  advantages:  (1)  it  provided 
the  military  coordinator  with  detailed  information  about  the  purpose  and  objectives; 
(2)  it  provided  participants  with  the  knowledge  of  what  the  researchers  expected  of 
them;  and  (3)  it  allowed  participants  to  consider  the  problem  they  would  address  at 
the  conference  in  advance.  The  third  point  is  important  in  that  it  provided  the 
opportunity  for  researchers  to  obtain  more  carefully  considered  judgments  from 
participants  than  would  be  likely  without  a  read-ahead  package. 
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FUTURE  PLANS 


Overview 

Year  1  of  this  research  effort  focused  on  how  the  intelligence  user  views  MI 
unit  effectiveness  and  usage  of  intelligence  products  by  users.  During  Year  2  of  the 
research  effort,  the  orientation  will  shift  to  the  intelligence  process  side  of  the  MI 
unit  effectiveness  picture.  This  next  phase  of  research  will  integrate  the  notion  of 
utility  measurement  with  diagnostic  procedures  to  identify  and  remedy  low  utility 
scores.  Year  3  of  the  research  effort  will  be  concerned  with  the  validation  of  the 
research  findings  from  Years  1  and  2. 

Measurement  Instrument  Development 

In  Year  1,  MAUA  was  demonstrated  as  a  viable  basis  for  the  development  of 
the  measurement  instrument.  Year  2  research  efforts  will  complete  the  development 
of  a  measurement  instrument  for  MI.  Completion  of  the  measurement  instrument 
requires  three  separate  tasks:  hierarchy  refinement,  hierarchy  application,  and 
refinement  of  the  external  factor  settings. 


ffl  to  iuhl  I  Mil  3  vIHMiMli'MI  iHil' 


The  Year  1  results  of  this  research  effort  have  produced  a  starting  base  for 
building  a  measurement  instrument  for  MI.  Oneimpor*-  outcome  of  the  first  year 
of  research  was  a  hierarchy  of  the  information  neeus  o*  intelligence  users  and 
weights  of  the  hierarchy  branches  for  two  different  situational  contexts.  Situational 
variables  have  been  identified  that  define  the  context  for  user  utility.  Further,  a 
number  of  utility  factors  have  been  identified  by  various  users  of  intelligence. 
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The  organization  of  information  needs  shown  in  the  hierarchy  must  be 
confirmed  by  additional  intelligence  users  and  weights  for  the  hierarchy  elicited. 

Our  approach  is  to  conduct  additional  decision  conferences  with  current  division 
staff  officers.  The  conferences  will  be  conducted  in  the  same  manner  as  the  MAUA 
conference  conducted  during  the  first  year.  Read-ahead  packages  will  be  developed 
that  contain  the  information  needs  hierarchy  and  revised  situational  factors  resulting 
from  the  MAUA  conference.  Both  the  hierarchy  and  situational  factors  will  be 
reviewed  and  suggested  modifications  elicited  from  conference  participants. 
Weights  will  be  elicited  from  the  conference  participants  on  the  hierarchy  for  two 
different  contexts. 


Information  Needs  Hierarchy:  Application 
The  information  needs  hierarchy  can  be  viewed  as  a  database  of  information 
needs  to  which  user  utility  can  be  applied.  Preliminary  trials  in  weighting  the 
hierarchy  by  field  personnel  demonstrates  that  we  can  discriminate  between 
peacetime  and  wartime  information  needs  as  well  as  between  users.  Viewing  the 
hierarchy  as  a  database  or  standard  might  be  termed  a  prescriptive  application  of  the 
hierarchy.  What  we  would  like  to  develop  is  a  profile  of  how  different  users  would 
weight  the  hierarchy  in  different  situations.  The  results  of  Year  2  will  determine 
how  many  profiles  might  be  needed  to  represent  output  utility  for  different  users  in 
different  situations.  For  the  training  environment  this  would  translate  to  the 
development  of  user  profiles.  Such  a  profile  could  be  viewed  as  bringing  a 
"mythical  G3"  into  the  classroom  to  serve  as  a  surrogate  user  for  the  evaluation  of 
student-generated  intelligence  output  The  profile  would  essentially  be  a  weighted 


78 


database  of  information  items  represented  by  the  information  needs  hierarchy. 
Further,  the  profile  would  be  specific  for  a  "mythical"  user  (G3,  G4,  FSC,  etc.)  as 
well  as  situationally  dependent  The  contents  of  student  intelligence  outputs  could 
then  be  assessed  by  instructors  against  the  profile  of  a  mythical  user.  In  this  way, 
students  could  receive  selective  feedback  about  the  potential  assessment  of  their 
outputs  from  the  perspective  of  the  possible  users  of  that  output  At  this  point  we 
cannot  be  certain  that  a  single  "mythical"  user  profile  for  each  user  type  will  be 
developed.  A  single  profile  is  likely  if  a  high  degree  of  consensus  exists  between 
user  weights  that  will  be  elicited  during  Year  2  site  visits.  If  normative  profiles 
cannot  be  achieved  due  to  lack  of  consensus,  then  multiple  user  profiles  will  be 
necessary. 

In  order  to  address  MI  performance,  in  the  classroom  or  field  exercises,  a 
scoring  approach  must  be  developed  that  includes  the  other  utility  factors  identified 
in  Year  1  research.  A  number  of  concerns  must  be  addressed  in  the  development  of 
a  scoring  approach.  For  one,  there  is  a  question  of  what  should  be  scored.  One 
alternative  is  to  score  every  output  on  an  individual  information  item  basis,  which 
could  be  very  labor-intensive  in  the  context  of  an  exercise  in  addition  to  the 
possibility  of  a  rater  "missing"  outputs.  An  alternative  is  to  assess  MI  performance 
on  the  basis  of  the  outputs  provided  to  users  over  a  period  of  time,  such  as  24 
hours.  If  this  approach  is  selected,  die  information  item  hierarchy  would  need  to  be 
weighted  from  the  time  period  perspective;  Le.,  the  relative  importance  of  receiving 
information  items  within  a  particular  time  period. 

The  next  step  in  scoring  development  is  to  refine  and  translate  the  full  set  of 
utility  factors  (e.g.,  accuracy,  timeliness)  into  qualitative  dimensions  for 
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assessment  The  prevailing  influence  on  the  direction  we  take  is  how  well  the  score 
can  be  used  to  diagnose  problems  in  the  intelligence  process.  A  score  in  and  of 
itself  has  little  value  to  the  intelligence  process.  If  a  unit  receives  a  low  score  in 
providing  high  utility  information  to  a  user,  the  important  question  to  answer  is 
"What  went  wrong"?  Given  the  ultimate  objective  of  determining  what  went 
wrong,  the  scoring  approach  developed  must  be  sufficiently  sensitive  to  serve  as  a 
pointer  into  problems  in  the  process. 

Context  Refinement 

The  situational  factors  identified  by  die  first-year  effort  include  five  separate 
factors  with  three  or  more  levels  to  each  factor.  The  permutation  of  factors  and 
levels  leads  to  a  rather  large  number  of  situational  cells  which  is  the  product  of: 
(echelon  levels)  x  (levels  of  conflict)  x  (number  of  users)  x  (mission  types)  x 
(application  levels)  =  3x3x5x3x4«*  540.  However,  since  our  effort  is 
concerned  with  the  development  of  only  one  echelon  this  reduces  the  number  to  36 
cells  (levels  of  conflict  x  mission  types  x  application  levels)  for  a  particular  user. 
While  the  numeric  reduction  is  an  improvement,  die  potential  requirement  for  36 
profiles  for  each  intelligence  user  at  division  echelon  might  be  viewed  as  impractical 
for  the  training  environment  It  is  necessary  that  we  attempt  to  look  for  ways  to 
reduce  the  set  of  situational  factors. 

The  set  of  situational  factors  will  be  reduced  in  two  ways.  First  using  SMEs 
within  the  project  team,  we  will  look  for  combinations  that  are  unrealistic.  A  cell 
such  as  peacetime/garrison/execution  is  not  likely  to  be  a  realistic  situation  to 
consider.  Each  combination  of  level  of  conflict,  mission,  and  application  will  be 
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assessed  for  realism;  if  the  combination  does  not  represent  a  possible  situation,  it 
will  be  dropped  from  consideration.  Next,  we  will  look  for  combinations  of  factors 
in  which  the  weighting  of  the  utility  hierarchy  does  not  vary  by  user.  Participants 
in  the  Fort  Leavenworth  conference  suggested  that  the  situational  factors  could  be 
combined  in  such  a  way  that  weightings  would  be  invariant.  'We  will  determine  if 
such  situational  combinations  can  be  made. 

Development  of  Process  Corollary  to  User  Utility 
The  research  effort  to  date  has  concentrated  on  the  development  of  a 
measurement  instrument  to  be  used  to  assess  intelligence  outputs  from  a  user 
perspective.  To  provide  guidance  to  the  intelligence  training  domain,  it  is  necessary 
to  go  beyond  the  evaluation  of  outputs  into  the  intelligence  process  itself.  In  other 
words,  guidance  must  be  developed  to  identify  the  source  of  problems  associated 
with  a  less  than  adequate  output  A  further  step  is  to  identify  the  remedies  that 
could  be  employed  to  improve  an  output  By  developing  this  type  of  guidance,  we 
will  not  leave  the  intelligence  community  with  the  idea  that  the  only  approach  to 
providing  users  with  the  highest  utility  output  is  to  produce  individual  outputs  for 
each  user.  Such  a  solution  would  be  untenable  from  an  intelligence  perspective 
because  of  resource  implications. 

In  the  second  year  of  the  research,  we  will  trace  user  utility  into  the 
intelligence  process.  A  number  of  steps  will  be  required,  including: 

•  Developing  a  model  of  the  intelligence  process 

•  Defining  the  supporting  intelligence  infrastructure 

•  Mapping  the  infrastructure  to  the  intelligence  process 
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•  Mapping  the  output  utility  factors  to  the  process 

•  Developing  remedies  for  utility  shortfalls. 

We  will  draw  on  the  intelligence  expertise  on  the  research  team  to  flesh  out 
each  step  identified  above.  It  is  important,  however,  that  active  Army  intelligence 
personnel  be  queried  to  validate  our  models.  While  our  models  of  the  processes  are 
likely  to  be  simplifications,  they  must  be  acceptable  to  the  intelligence  community  as 
reasonable.  The  research  team  will  work  with  senior  Army  officers  at  the 
Intelligence  Center  and  School  in  an  attempt  to  capture  the  current  thinking  of 
intelligence  doctrine  in  the  process  model  development  In  addition,  interviews 
with  intelligence  personnel  could  be  conducted  in  conjunction  with  the  decision 
conferences  held  in  the  second  year  of  the  research.  Individuals  to  be  interviewed 
would  include  active  Army  personnel  currently  serving  in  roles  such  as  G2,  G2 
Operations,  MI  battalion  commander,  Chief  of  All  Source  Production  Section, 

Chief  of  Collection  Management  &  Dissemination,  and  Senior  Order  of  Battle 
Technician. 


Approaches  to  Validation 

The  validation  of  the  research  findings  of  this  effort  is  critical  to  the  ultimate 
success  ?nd  implementation  of  a  method  for  measuring  the  effectiveness  of  an  MI 
unit.  Validation  is  required  of  the  measurement  instrument  and  the  diagnostic 
procedure  for  tracing  user  utility  into  the  intelligence  process. 


b2 


Validation  of  Measurement  Instrument 
Any  single  methodological  approach  runs  the  risk  of  failing  to  identify  some 
key  factors  and  developing  only  a  partial  understanding  of  the  problem.  To  avoid 
this  possibility,  a  second  method  will  be  employed  during  the  second  year  of  the 
research  to  serve  as  a  validation  of  the  findings  of  the  MAUA  approach. 

Two  methods  have  been  considered  that  could  provide  the  necessary 
validation.  One  method,  a  factor  analytic  approach,  is  a  survey-based  approach. 
This  approach  uses  self-administered  pencil  and  paper  survey  instruments  to 
structure  the  problem  space  and  develop  a  metric.  With  this  approach,  survey 
respondents  are  asked  to  rate  several  MI  outputs  on  a  list  of  potential  attributes. 

The  data  collected  are  evaluated  using  factor  analytic  techniques  to  determine  the 
higher  level  factors  (e.g.,  description  of  the  battlefield  area,  description  of  enemy 
situation)  that  determine  the  conceptual  structure  of  the  problem  area.  Using  the 
results  of  the  initial  survey,  a  final  version  of  the  rating  scales  would  be  developed 
that  could  be  used  to  quantitatively  assess  the  components  of  factors  of  output 
utility.  A  second  survey  would  be  used  to  determine  how  the  relative  importance  of 
the  factors  change  depending  on  the  user  and  the  situation. 

The  factor  analytic  approach  represented  an  attractive  possibility  since  it  is  a 
powerful  methodology  from  a  statistical  standpoint  However,  this  approach  was 
rejected  for  two  reasons:  survey  participant  requirements  and  time.  The  number  of 
individuals  required  for  the  factor  analytic  approach  was  extremely  high  and  some 
requirements  could  not  be  met  with  the  active  Army  personnel  within  the  continental 
United  States.  An  additional  problem  was  the  fan  that  such  an  extensive  survey 
could  not  be  accomplished  within  the  second  year  of  the  research  effort 
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Policy  capturing  has  been  selected  as  the  most  viable  approach  to  the  validation 
of  the  measurement  instrument.  Policy  capturing,  a  proven  method  in  the  area  of 
social  judgment  theoiy ,  does  not  have  the  problems  inherent  in  the  factor  analytic 
approach  in  terms  of  sample  size  requirements.  Most  importantly,  policy  capturing 
offers  an  approach  to  validation  of  the  measurement  instrument  that  focuses  on  the 
judgment  of  intelligence  users. 

Policy  Capturing:  Approach 

Policy  capturing  appears  to  be  a  viable  technique  for  the  validation  of  the 
measurement  instrument  Policy  capturing  uses  multiple  regression  analysis  to 
develop  a  statistically  based  model  indicating  the  relative  importance  of  different 
attributes  when  predicting  an  individual’s  judgments.  The  attributes  represent  the 
independent  variables  in  the  multiple  regression  analysis;  the  individual’s  judgments 
represent  the  dependent  variable. 

Policy  capturing  can  be  used  in  a  number  of  ways  for  validation  purposes. 

The  information  needs  hierarchy,  when  weighted,  predicts  the  utility  of 
informadon.  Policy  capturing  could  be  used  to  validate  these  predictions. 
Experimentally,  policy  capturing  could  be  used  at  any  point  in  the  hierarchy  to 
validate  predicted  utility  of  information  items.  This  could  be  accomplished  by 
presenting  subjects  with  intelligence  outputs,  specifically  designed  to  represent 
different  values  of  information  items.  In  other  words,  hypothetical  intelligence 
outputs  would  be  developed  to  represent  different  values  predicted  from  the  MAUA 
hierarchy.  An  alternative  approach  would  be  to  present  subjects  with  the  semes  of 
a  "fictional"  unit's  performance.  For  example,  a  fictional  unit  may  have  received  a 


high  score  on  "description  of  the  battlefield  area"  and  "description  of  enemy 
situation"  and  a  low  score  on  "description  of  enemy  CO  As."  The  weighted  MAUA 
predicts  how  this  combination  of  scores  is  likely  to  be  assessed  for  a  particular  user 
in  a  particular  context  In  either  case,  subjects  would  be  asked  to  provide  an  overall 
judgment  of  the  output  or  the  unit's  performance.  The  attributes  under  study 
would  have  low,  medium,  or  high  values  in  the  case  of  outputs.  Performance 
attribute  values  would  be  randomly  generated.  The  correlations  between  attribute 
values  over  all  outputs  or  performances  approach  zero  so  that  we  have  statistical 
independence  among  the  attributes  for  the  multiple  regression  analysis.  The  linear 
model  for  analyzing  an  individual  judgment  is  represented  as  follows: 

m 

Yij  =  Tbikxjk  +  ci  +  cij 
k=l 

where 

Yjj  *  The  judgment  of  individual  i  for  profile  j  (e.g.  a  hypothetical  output 
or  a  hypothetical  unit's  performance  score) 

m  «=  The  number  of  attributes 

bik,  k  =  1,  m,  the  •  ’ue  on  attributes,  raw  score  regression  weight  on 
attribute  k  lilWtW.  when  normalized  and  divided  by  the  sum  of 
normalized  weights  it  indicates  the  relative  importance  of  the 
attribute  for  the  individual's  judgments) 

xjk  k  ■  1,  tn,  the  value  of  attribute  k  for  profile  j 

ci  Constant  term  for  individual  i 

cij  Is  the  residual  error  from  the  model  of  individual  i  for  profile  j. 
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Consequently,  one  can  determine  the  relative  weight  a  participant  places  on 
each  attribute  when  evaluating  the  outputs  or  units  by  examining  die  (normalized) 
beta  weights.  The  larger  the  beta  weight,  the  more  important  the  attribute. 
Moreover,  the  correlation  between  an  individual's  actual  judgment  and  the  judgment 
predicted  by  the  multiple  regression  model  (Le.,  die  multiple  correlation  coefficient, 
R)  Represents  a  quantitative  measure  of  the  quality  of  the  model.  Low  values  of  R 
(e.g.,  <  .5)  would  indicate  the  inadequacy  of  the  model  because  of  the  following 
reasons:  (1)  the  individual  is  inconsistent  in  using  the  model  to  evaluate  the 
products;  or  (2)  the  individual  is  using  a  different  type  of  model  to  evaluate  the 
products;  or  (3)  we  failed  to  capture  other  important  variables  that  die  individual  is 
using  to  evaluate  the  products. 

With  policy  capturing  we  can  generate  relative  weights  for  user  types  (e.g., 

G3,  G2,  etc.).  An  alternative  or  additional  analysis  could  regress  the  independent 
variable  values  on  the  averaged  judgments  of  the  participants  in  each  group. 
Importantly,  the  policy  capturing  data  may  be  used  to  validate  the  MAUA  weights 
by  applying  the  weights  to  the  products  to  get  predicted  values  for  the  products  and, 
then,  correlating  the  predicted  values  with  the  actual  (averaged)  judgments.  High 
correlations  would  validate  the  group  MAUA  weights;  low  correlations  would 
suggest  problems  analogous  to  those  for  low  R  values. 

Rolicv  Capturing;  Advantages 

Policy  capturing  has  been  extensively  used  to  generate  quantitative  measures  of 
interpersonal  agreement  and  interpersonal  understanding,  a  capability  applicable  to 
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our  objective.  Policy  capturing  offers  a  number  of  advantages  in  this  research 
context,  including: 


•  Policy  capturing  builds  on  the  MAUA  approach  to  metric  development  by 
using  the  factors  already  identified. 

•  Policy  capturing  will  be  implemented  by  having  participants  evaluate  MI 

outputs  or  scores  of  fictious  units,  that  have  predicted  scores  from  the 
MAUA-based  utility  metric. 

•  The  results  of  policy  capturing  are  statistically  based  models  indicating  the 
relative  importance  of  each  MI  attribute  on  the  participant’s  judgment, 
facilitating  a  quantitative,  multimethod  comparison  between  the  relative 
weights  in  the  MAUA  hierarchy  and  those  generated  by  policy  capturing  for 
different  users  in  different  contexts. 

•  The  MAUA  weights  may  be  applied  to  the  attribute  values  for  each  output  or 

unit  performance  to  derive  a  set  of  MAUA-based  predicted  judgments, 
thereby  using  the  correlations  between  (a)  MAUA-based  predicted 
judgments  and  the  actual  judgments,  (b)  the  policy  capturing-based 
predicted  judgments  and  the  actual  judgments,  and  (c)  the  MAUA-based  and 
policy  capturing-based  predicted  judgments  for  validation  purposes. 

Validation  of  Diagnostic  Procedure 

The  validation  of  the  diagnostic  procedure  requires  the  validation  of  the 
numerous  components  of  the  procedure.  First,  we  must  validate  the  process  and 
infrastructure  models  and  the  mapping  of  the  two.  This  will  be  accomplished  in 
two  ways.  First,  our  strawman  models  will  be  presented  to  and  modified  by  senior 
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intelligence  personnel  at  the  Intelligence  Center  and  School.  In  this  way,  the  model 
will  be  validated  for  doctrinal  soundness.  The  revised  model  will  be  reviewed  by 
intelligence  personnel  at  FORSCOM  divisions.  The  reason  for  validation  of  the 
models  by  different  groups  is  that  the  diagnostic  procedure  needs  to  be  applicable  to 
the  classroom  as  well  as  the  field. 

The  actual  diagnostic  procedure  of  tracing  a  low  score  to  a  point  in  die  process 
must  be  validated.  One  approach  to  the  validation  is  war  gaining  low  process 
scores  with  intelligence  staffs  to  determine  the  consistency  of  the  pointers  into  the 
process.  This  procedure  could  be  used  with  division  intelligence  staff  personnel 
such  as  the  G2,  MI  Battalion  Commander,  G2  Operations,  Chief  of  All  Source 
Production  Section,  Chief  of  Collection  Management  &  Dissemination,  and  the 
Senior  Order  of  Battle  Technician.  Personnel  could  be  presented  with  a  series  of 
low  process  scores  and  asked  to  diagnose  the  process  and  isolate  the  problem. 

The  remedies  developed  for  utility  shortfalls  must  also  be  validated.  In  Year  2 
researchers  will  investigate  a  number  of  possible  approaches  to  the  validation  of 
remedies.  Since  remedies  may  have  a  cost  of  implementing  associated  with  them, 
we  will  consider  techniques  in  the  area  of  cost-benefit  analysis  as  offering  a 
possible  validation  approach. 

PcTfermmi  Based  validation 

While  the  validation  of  the  methodology  is  important,  it  will  not  answer  one 
question:  "Does  increasing  the  utility  of  intelligence  outputs  improve  the 
performance  of  the  operational  user?"  This  is  essentially  a  bottom  line  question  to 
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consider  and  the  answer  requires  a  performance  based  evaluation  of  intelligence 
users. 

The  evaluation  is  likely  to  be  concerned  with  individual  performance  of 
intelligence  users  rather  than  on  team  performance,  such  as  a  staff  during  an 
exercise.  The  reason  for  not  focusing  on  team  performance  is  that  there  are  a 
number  of  variables  in  team  performance  drat  could  not  readily  be  controlled.  In  a 
staff  exercise,  one  could  not  conclude  that  the  quality  of  intelligence  "made  all  the 
difference"  in  the  results  of  the  exercise.  Focusing  on  individual  performance, 
while  more  artificial,  is  more  controllable  and  hence  desirable. 

The  Army  Research  Institute  is  in  the  process  of  developing  an  experimental 
laboratory  to  study  division  level  command  and  control  decision  making.  This 
facility,  called  the  Experimental  Development,  Demonstration,  and  Integration 
Center  (EDDIC),  is  located  at  Fort  Leavenworth  and  would  allow  the  experimental 
control  of  information  to  subjects.  Importantly,  EDDIC  will  produce  a  wide  range 
of  dependent  measures  including  comparisons  with  school  book  solutions,  time, 
information  use  and  flow. 

Practicality  of  the  Measurement  Methodology 

There  are  some  practical  concerns  that  need  to  be  addressed  in  the  development 
of  the  measurement  methodology.  Practical  concerns  in  some  ways  reflect  the 
conflicting  nature  of  applying  research  to  the  military  setting.  On  the  one  hand, 
researchers  may  strive  for  developing  a  rigorous  methodology  that  meets  the 
standards  of  an  academic  community.  On  the  other  hand,  the  military  setting  may 
appreciate  the  standards  but  have  somewhat  different  concerns.  It  is  reasonable  to 
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assume  that  the  MI  setting  would  prefer  a  methodology  that  can  be  used  by  military 
instructors  rather  than  usable  only  by  scientific  observers.  Further,  the  MI  setting 
is  likely  to  be  concerned  with  how  rapidly  they  may  provide  feedback  to  a  unit  on 
it's  performance. 

The  measurement  methodology  must  be  easily  applied  and  not  require  an 
inordinate  expenditure  of  resources  to  apply.  For  example,  a  methodology  that 
requires  a  large  number  of  data  collectors  might  not  be  practical.  Even  less  practical 
would  be  a  measurement  methodology  that  requires  extensive  "off-line"  analysis. 
We  must  strive  for  a  methodology  that  can  find  the  problems  in  MI  effectiveness, 
vis-a-vis  low  scores,  and  diagnose  the  problem  and  suggest  remedies  in  near  real 
time.  Optimally,  the  evaluation  of  one  twenty-four  hour  assessment  period  would 
lead  to  improved  performance  during  the  next  twenty-four  hour  assessment  period. 
Such  near  real  time  evaluative  feedback  will  be  an  objective  of  the  research. 

A  related  practical  concern  deals  with  the  manner  in  which  scoring  will  be 
accomplished.  Scoring  needs  to  be  handled  at  the  lowest  attribute  level  of  the 
hierarchy  and  there  are  a  number  of  separate  scoring  dimensions,  this  would 
translate  to  a  rather  large  number  of  scoring  questions  for  evaluation.  This  would 
raise  a  practical  concern  in  implementation.  Because  of  practical  concerns  about 
time  required  to  administer  scoring  questions  and  particularly  time  intrusions  on 
users,  we  are  approaching  the  scoring  methodology  on  the  basis  of  high 
discriminability  scoring  for  diagnosis  as  opposed  to  comprehensive  evaluation. 
Other  implementation  options  such  as  automated  support  could  substantially  reduce 
adminstration  time  and  shorten  the  feedback  loop. 
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Expected  Products  and  Milestones 

This  project  is  developing  a  methodology  for  measuring  the  effectiveness  of 
MI.  As  envisioned,  this  methodology  may  be  used  to  assess  students  as  well  as  MI 
units  in  an  exercise.  To  reach  our  objective  a  number  of  milestones  are  expected  to 
be  accomplished  in  Year  2  as  shown  in  Figure  11.  The  products  that  will  be 
produced  by  the  conclusion  of  Year  2  include: 

•  Normative  profiles  for  classroom  use 

•  Measurement  instrument  for  scoring  MI  unit  performance  from  a  user  utility 
perspective 

•  Methodology  for  diagnosing  low  scores  in  MI  unit  performance 

•  Identification  of  potential  remedies  for  diagnosed  problems 

•  Technical  report  of  Year  2  findings. 

The  expected  results  of  Year  3  include: 

•  Validation  of  profiles  (policy  capturing  for  validatiing  MAUA  predictions) 

•  Validation  of  diagnostic  procedure 

•  Validation  of  remedies 

•  Documentation  for  implementing  the  measurement  methodology. 
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Appendix  A 


INTERVIEW  PROTOCOL 


INTRODUCTION 

We  are  conducting  a  study  into  the  usefulness  of  military  intelligence 
outputs.  The  study  is  aimed  at  finding  out  how  to  make  sure  command  staff 
personnel  get  the  most  useful  information  possible.  The  ultimate  impact  of 
this  research  will  be  directed  toward  increasing  the  utility  of  intelligence 
you  receive.  What  we  need  to  learn  from  you  is  how  and  when  information 
needs  differ. 

During  this  interview,  we'll  ask  you  to  think  about  two  different 
scenarios.  In  each  scenario,  we'll  ask  you  what  information  is  most 
important,  what  makes  good  intelligence,  and  how  intelligence  can  be  made 
more  useful  to  you. 

We  prefer  to  keep  the  contents  of  this  interview  unclassified.  If  you 
don't  mind,  we  would  like  to  tape-record  the  interview  for  our  own  use.  Of 
course,  anything  you  say  here  is  confidential.  No  results  of  this  interview 
will  be  used  in  any  way  that  could  identify  you. 


BACKGROUND 

1.  Before  we  begin,  could  you  please  give  us  a  brief  description  of  the 
types  of  division  (or  corps)  related  experience  you  have  had? 


2.  What  are  (were)  your  primary  and  alternate  specialties? 


A-l 


Let's  begin  by  setting  the  scenario.  Imagine  you  are  a  [Cmdr,  G-3,  G-3 
Ops,  or  G*3  Plans]  in  a  division  in  Central  Europe  in  a  peacetime  situation. 
Since  our  focus  is  on  information  needs,  let's  consider  your  day-to-day  job 
as  a  context.  Would  you  characterize  the  potential  threat  you  face  as  high- 
intensity  conflict? 


[If  the  answer  is  no,  ask:  Would  you  be  involved  in  contingency 
operations  involving  anti -terrorism?] 


To  further  specify  information  needs,  let's  consider  a  specific  task 
that  you  perform  as  part  of  your  job.  We  would  like  to  select  a  task  in 
which  you  use  intelligence  information  in  the  performance  of  the  task. 
Doctrine  describes  a  large  number  of  tasks  performed  by  the  G-3.  We  would 
like  you  to  focus  on  the  task  maintaining  a  current  operation  estimate  of 
the  situation. 

Do  (or  Have)  you  have/had  experience  performing  this  task? 

[If  no:  Do  (or  Have)  you  have/had  experience  with: 

o  preparing  and  publishing  operations  plans 
o  preparing  and  publishing  frag  orders 

o  recommending  prioroties  for  allocating  resources  of  the 
command. ] 

[If  no:  Can  you  think  of  another  task  that  you  have  experience  with 

that  relies  on  intelligence?  What  is  the  task?] 


Do  you  feel  comfortable  with  this  scenario? 

[If  no  on  a  specific  aspect,  e.g.,  definition  of  the  mission:  Within 
the  general  scenario  outlined,  what  do  you  think  is  the  most  likely 
(mission)?] 


1.  With  this  context  in  mind,  how  often  do  you  (or  have  you)  perform(ed) 
this  task? 


2.  What  are  the  first  things  you  want  to  know  from  intelligence? 


o  PROBE:  What  do  you  need  to  know  next? 


[Clarify  and/or  restate  their  answers  at  this  point.] 
o  PROBE:  What  do  you  need  to  know  about  . ? 


[If  they  respond  METT-T,  PROBE:  is  the  minimum  information  about  the 
(mission  or  enemy  or  terrain  or  troops  or  time  available)  that  you 
need  to  know?] 


3.  What  factors  are  most  important  in  making  good  MI  intelligence 
information? 
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o  PROBE:  Any  other  factors? 


[Clarify  and/or  restate  the  answers] 

[PROBE  for  more  detail  regarding  the  general  factors  provided: 

*  How  do  you  determine  reliability? 

-  What  do  you  mean  by  accuracy? 

-  What  do  you  mean  by  timeliness? 

-  What  would  you  tolerate  in  accuracy  reduction  in  order  to  get 
more  timely  information?] 


In  specific,  what  can  be  done  to  improve  the  usefulness  of  intelligence 
that  you  receive? 


o  PROBE:  Are  there  any  other  things  that  can  be  done  to  improve  the 
usefulness? 


[Clarify  and/or  restate  the  answers.] 


SCENARIO  2 

Now  let's  continue  talking  about  the  same  situation,  but  change  the 
context.  You  are  in  the  same  position,  same  unit,  (same  place),  with  the 
same  task,  but  rather  than  day-to-day  operations  in  a  peacetime  situation, 
imagine  that  you  are  in  [high-intensity/terrorist  hostilities  or  pre¬ 
hostilities].  [By  pre-hostility  we  mean  that  the  command  has  been  placed  in 
an  advanced  state  of  readiness  or  placed  on  alert.  In  other  words,  DEFCON 
status  has  been  changed  from  DEFCON  5  to  DEFCON  4.]  We  would  like  to  know 
how  your  intelligence  needs  would  change. 


1.  Does  the  frequency  of  performing  the  task  change  in  this  context? 


2.  In  day-to-day  operations,  you  said  that  you  wanted  .  first, 

.  information  next,  .  [Fill  in  the  blanks  with  his  responses 

to  the  first  scenario]. 

Does  the  information  you  need  first  from  intelligence  differ  in  this 
context  over  day-to-day  operations? 


o  PROBE:  Any  other  changes? 


o  [If  they  say  nothing  will  change  PROBE:  You're  saying  that  you 
still  need . first  and  then  . ] 


3.  Are  different  factors  important  in  making  good  intelligence 
information?  [Review  factors  identified  in  Scenario  1  if  necessary.] 


o  PROBE:  Any  other  changes  in  the  factors? 


o  [If  they  say  nothing  will  change  PROBE:  You  're  saying  that  . 

are  still  the  most  important  factors  in  a  pre-hostility 
environment?] 


4.  Would  you  do  anything  differently  to  improve  the  usefulness  of 
intelligence  that  you  receive?  [Review  responses  to  Scenario  1  if 
necessary. ] 


o  PROBE:  Would  you  do  anything  else  differently? 


OPTIONAL  QUESTION: 


What  makes  a  good  G-2?  What  are  the  attributes  of  a  good  G-2? 


PHASE  II 

INTERVIEW  PROTOCOL 


INTRODUCTION 


We  are  conducting  a  study  Into  the  usefulness  of  military  intelligence 
outputs.  The  ultimate  impact  of  this  research  will  be  directed  toward 
increasing  the  utility  of  intelligence  users  receive.  This  interview  is 
part  of  the  second  phase  in  a  series  of  Interviews  aimed  at  uncovering  what 
makes  intelligence  outputs  useful  to  command  staff  personnel.  In  the  first 
phase,  we  tried  to  determine  how  and  when  information  needs  differ.  In  this 
phase,  we  want  to  further  define  what  makes  useful  intelligence. 

We  would  like  to  keep  the  contents  of  this  Interview  unclassified.  If 
you  don't  mind,  we  would  also  like  to  tape-record  the  interview  for  our  own 
use.  Of  course,  anything  you  say  here  is  confidential.  No  results  of  this 
interview  will  be  used  in  any  way  that  could  identify  you. 


BACKGROUND 


1.  Before  we  begin.  I'd  like  to  go  over  your  related  experience  again. 
You  were  a(n)  (POSITION)  at  the  (UNIT)  level.  At  that  time  you 
were  a  (RANK)  . 


2.  Your  primary  and  alternate  specialties  were 
_ ,  respectively. 


and 
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CONTEXT  A 


As  indicated  in  the  handout  you  received  ahead  of  time,  we  identified 
several  factors  in  previous  interviews  that  are  important  in  making  good 
intelligence  information.  While  we  received  sufficient  clarification  on 
some  of  the  factors,  several  of  the  factors  are  still  somewhat  unclear.  In 
this  interview,  we'd  like  to  obtain  further  clarification  on  these  factors 
from  you. 

We'd  like  to  discuss  these  factors  in  a  general  scenario.  Imagine  you 
are  a  [Cmdr,  G-3,  G-S  Ops,  or  G-3  Plans]  in  a  division  in  Central  Europe 
(high-intensity  conflict  threat).  You  are  in  a  peacetime  situation.  Do  you 
feel  comfortable  with  this  scenario? 


With  this  context  in  mind,  let's  discuss  some  of  the  factors 
identified. 


1.  In  the  earlier  interviews,  clarity  was  a  factor  identified  as  important 
in  making  good  intelligence  information.  What  do  you  mean  by  clarity? 

•  easily  understood  by  user 

•  in  operational  terms 


2.  Another  factor  identified  as  important  for  good  Intelligence  was 
accuracy  or  credibility.  What  do  you  mean  by  accuracy? 

•  likelihood  of  information  being  true 
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How  is  accuracy  determined? 


•  information  that  is  confirmed  by  multiple  sources 


Timeliness  was  identified  as  extremely  important  for  good  intelligence. 
If  you  had  to  choose  between  timeliness  and  accuracy  which  would  you 
choose? 


3. 


A  third  important  factor  identified  was  reliability, 
by  reliability. 

•  reliable  information  is  information  that  can 


What  do  you  mean 
be  trusted 


How  do  you  determine  reliability? 

•  source  of  intelligence 

•  prevous  experience  with  source  of  intelligence 


4.  The  earlier  interviews  indicated  that  good  intelligence  information  is 
filtered  or  processed.  What  do  you  mean  by  filtered/processed  informa¬ 
tion? 

•  screening  information 
t  summarizing  information 
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If  information  should  be  screened,  what  should  be  screened  out? 
•  information  not  relevant  to  the  unit's  objective(s) 


5.  The  earlier  interviews  also  Indicated  that  it  is  important  for  the 
information  to  be  Interpreted.  What  do  you  mean  by  Information  that  is 
interpreted? 

•  assessed  for  its  impact 

•  conclusions  drawn 


6.  A  final  factor  identified  in  the  earlier  interviews  as  important  in 
making  good  intelligence  information  is  relevance.  What  do  you  mean  by 
relevance?  Relevant  to  what? 

t  related  to  unit's  mission/objective 


7.  [Discuss/clarify  factors  added  to  the  list  in  Section  1  of  the  handout 
given  prior  to  the  interview.]  What  do  you  mean  by  _ ? 
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Now  let's  continue  talking  about  the  same  situation,  but  change  one 
thing.  Rather  than  a  peacetime  situation,  imagine  that  you  are  in  high- 
intensity  hostilities.  We  would  like  to  know  how  your  intelligence  needs 
would  change. 


1.  Let's  look  at  Section  3  of  the  handout  and  see  if  any  of  the  factors 
change  in  importance  in  this  context. 


[Review  Section  3  of  the  handout.  Discuss/clarify  any  factors  that 
changed  in  importance  from  peacetime  to  active  hostilities.] 


2.  Are  different  factors  important  in  making  good  intelligence  informa¬ 
tion?  Any  other  changes  in  the  factors. 


3.  [Review  Section  2  of  the  handout.  Discuss/clarify  any  changes  from 
peacetime  to  active  hostilities.] 


MILITARY  INTELLIGENCE  UNIT  PERFORMANCE  PROJECT 
READ-AHEAD  PACKAGE  FOR  PHASE  II  INTERVIEWS 


SECTION  1 


The  following  list  contains  factors  identified  in  previous  interviews  as 
important  in  making  good  intelligence  information.  Each  factor  is  followed 
by  a  definition  based  on  the  first  round  of  interviews.  In  order  to  verify 
that  these  factors  were  defined  correctly  and  are  indeed  important  in  making 
good  intelligence  information,  we  would  like  you  to  review  this  list. 
Please  expand,  clarify,  or  correct  these  definitions  as  you  deem  appro¬ 
priate.  If  you  would  like  to  add  any  factors  please  do  so  in  the  spaces 
provided  at  the  end  of  the  list.  We  will  go  over  your  comments  more  exten¬ 
sively  in  the  upcoming  interview. 


1.  Timeliness  -  does  the  information  get  to  the  user  in  time;  timely 
information  is  information  that  the  user  receives  in  time  to  allow  an 
appropriate  response. 


2. 


Currency  -  age  or  perishability  of  the  information;  current  information 
is  information  that  still  accurately  represents  the  present  situation. 


3.  Completeness  -  specificity;  how  specific  or  detailed  the  information 
is;  information  with  the  sufficient  level  of  detail. 


A.  Conciseness  -  succinctness;  without  unnecessary  detail;  concise  infor¬ 
mation  is  succinct  and  to  the  point. 


5.  Clarity  -  information  is  in  terms  (i.e.,  operational  or  tactical  terms) 
easily  understood  by  the  user. 


6.  Graphical  -  Information  is  presented  in  graphic  form  to  the  extent 
possible. 
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7.  Accuracy  -  the  Information  presented  is  factually  correct. 


8.  Credibility  -  the  extent  to  which  information  is  confirmed;  the 
probability  that  the  information  is  true. 


9.  Source  Reliability  -  information  is  from  sources  that  can  be  trusted; 
reliability  is  based  on  the  source  of  the  intelligence  and  previous 
experience  with  the  source. 


10.  Filtered/Processed  -  information  that  has  been  screened  or  summarized. 


11.  Interpreted  -  information  is  assessed  in  terms  of  its  impact; 
information  for  which  conclusions  are  drawn  concerning  its  impact. 


12.  Relevance  -  information  that  is  explicitly  related  or  pertinent  to  the 
unit's  mission/objective. 


13. 


14. 


15. 
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SECTION  2 


The  following  items  were  Identified  in  previous  interviews  as  the  type  of 
information  needed  first  from  intelligence.  Please  review  this  list  of 
information  types  and  add,  delete,  modify,  or  annotate  the  items  based  on 
your  experience  as  a  user  of  intelligence.  Additionally,  for  each  item 
check  the  box  that  best  indicates  how  important  you  feel  this  type  of 
information  is  in  peacetime  situation.  (Please  check  one  box  per  item.) 


Part  A  -  Peacetime 


Not 

Important  Moderately  Extremely 

At  All  Important  Important 


1.  Enemy  Capabilities:  [1] 

a.  Unit  type  [1] 

b.  Size  [1] 

c.  Reinforcements  [1] 

d.  Intelligence  efforts  [1] 

e.  Leadership  (!) 

f.  Morale  [1} 

g.  Weaknesses/vulnerabilities  [1] 

2.  Location  of  Enemy  [1] 

3.  Enemy  Movement/Activity  [1] 

4.  Enemy  Order  of  Battle  [1] 

5.  Changes  in  Enemy  Posture  [1] 

6.  Political  Climate  [1] 

7.  Terrain  II] 

8.  Weather  [1) 

9.  [1] 

10*  _  [1] 

n. _  [i] 

12.  _  Cl] 


12) 

[3) 

[4] 

[5] 

[6] 

[7) 

12) 

[3) 

[4] 

[5) 

[6] 

[7] 

[2) 

[3) 

[4] 

[5) 

[6] 

[7] 

12) 

[3) 

[4) 

[5) 

[6) 

[7] 

[2] 

[3) 

[4) 

[5] 

[6] 

[7) 

[2) 

[3] 

[4] 

[5] 

[6] 

[7) 

[2] 

[3) 

[4] 

[5] 

[6] 

[7] 

[2) 

[3) 

[4) 

[5] 

16) 

[7] 

[2) 

[3) 

[4) 

[5) 

[6] 

17) 

[2] 

[3) 

[4) 

[5) 

[6] 

[7] 

[2) 

[3) 

[4) 

[5) 

[6] 

[7) 

[2) 

[3) 

[4) 

15) 

[6] 

[7) 

[2) 

[3) 

[4) 

15) 

[6] 

[7] 

[2) 

13) 

[4) 

[5) 

[6) 

[7] 

[2] 

[3) 

[4) 

[5) 

[6) 

[7] 

[2] 

[3] 

[4) 

[5] 

16) 

[7] 

[2) 

[3) 

[4] 

[5] 

16) 

[7) 

[2] 

[3) 

[4) 

E5] 

[6) 

[7] 

[2) 

[3] 

[4) 

[5) 

[6) 

[7) 
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This  section  asks  how  Important  you  think  the  information  types  are  in 
hostilities.  For  each  item  check  the  box  that  best  indicates  how  important 
you  feel  this  type  of  information  is  in  active  hostilities.  (Please  check 
one  box  per  item.) 


Not 

Important  Moderately  Extremely 

At  All  Important  Important 


1.  Enemy  Capabilities:  [1] 

a.  Unit  type  [1] 

b.  Size  [1] 

c.  Reinforcements  [1] 

d.  Intelligence  efforts  [1] 

e.  Leadership  [1] 

f.  Morale  [1] 

g.  Weaknesses/vulnerabilities  [1] 

2.  Location  of  Enemy  [1] 

3.  Enemy  Movement/Activity  [1] 

4.  Enemy  Order  of  Battle  [1] 

5.  Changes  in  Enemy  Posture  [1] 

6.  Political  Climate  [1] 

7.  Terrain  [1] 

8.  Weather  II] 

9.  _  [1] 

10.  _  [1] 

11.  _  m 

[13 


[2] 

[3] 

[4] 

[5] 

[6] 

17] 

[2] 

[3] 

[4] 

[5] 

[6] 

[7] 

[2] 

[3] 

[4] 

[5] 

[6] 

[7] 

[2] 

13] 

[4] 

15] 

16] 

[7] 

[2] 

[3] 

[4] 

[5] 

[6] 

[7] 

[2] 

[3] 

[4] 

[5] 

[6] 

[7] 

[2] 

[3] 

[4] 

[5] 

[6] 

17] 

[2] 

[3] 

[4] 

[5] 

[6] 

[7] 

[2] 

[3] 

[4] 

[5] 

16] 

[7] 

12] 

[3] 

[4] 

[5] 

[6] 

[7] 

12] 

13] 

[4] 

15] 

[6] 

[7] 

12] 

13] 

[4] 

[5] 

[6] 

17] 

[2] 

[3] 

[4] 

[5] 

[6] 

[7] 

12] 

[3] 

[4] 

[5] 

[6] 

[7] 

12] 

13] 

14] 

[5] 

[6] 

17] 

[2] 

[3] 

[4] 

[5] 

[6] 

[7] 

[2] 

[3] 

[4] 

[5] 

[6] 

17] 

12] 

13] 

[4] 

15] 

16] 

[7] 

[2] 

[3] 

[4] 

[5] 

[6] 

[7] 

12. 


SECTION  3 


This  section  asks  how  important  you  think  the  previously  identified  factors 
are  in  making  good  intelligence  information  in  peacetime.  For  each  item, 
check  the  box  that  best  indicates  how  important  you  feel  the  factor  is  in  a 
peacetime  situation.  (Please  check  one  box  per  item.) 

Part  A  -  Peacetime 


Not 

Important 


At  All 

1.  Timeliness 

[13 

[23 

2.  Currency 

[13 

[23 

3.  Completeness 

[13 

[23 

4.  Conciseness 

[13 

[23 

5.  Clarity 

[13 

[2] 

6.  Graphical 

[1] 

[23 

7.  Accuracy 

[13 

[23 

8.  Credibility 

[1] 

[23 

9.  Reliability 

[13 

[23 

10.  Filtered/Processed 

[l] 

[23 

11.  Interpreted 

[U 

[23 

12.  Relevance 

[13 

[23 

13. 

[13 

[23 

14. 

[13 

[23 

15. 

[13 

[23 

Moderately 

Important 

Extremely 

Important 

[33 

[4] 

[5] 

[6] 

[7] 

[33 

[4] 

[5] 

[6] 

[7] 

[33 

[43 

[53 

[63 

[73 

[3] 

[4] 

[5] 

[6] 

[7] 

[33 

[43 

[5] 

[6] 

[7] 

[3] 

[4] 

[5] 

[6] 

[73 

[33 

[43 

[53 

[6] 

[7) 

[33 

[4] 

[5] 

[6] 

[7] 

[3] 

[4] 

[5] 

[6] 

[7] 

[33 

[4] 

[5] 

[6] 

[73 

[33 

[4] 

[53 

[6] 

[7] 

[3] 

[4] 

[5] 

[63 

[7] 

[33 

[43 

[53 

[6] 

[73 

[33 

[4] 

15] 

[6] 

[73 

[33 

[43 

[5] 

[6] 

[7] 
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Part  B  -  Active  Hostilities 


This  section  asks  how  important  you  think  various  factors  are  in  making  good 
intelligence  information  in  hostilities.  For  each  item  check  the  box  that 
best  indicates  how  important  you  feel  the  factor  is  in  active  hostilities. 
(Please  check  one  box  per  item.) 


Not 

Important 


At  All 

1. 

Timeliness 

m 

[2] 

2. 

Currency 

11] 

[2] 

3. 

Completeness 

m 

[2] 

4. 

Conciseness 

11] 

[2] 

5. 

Clarity 

U3 

[2] 

6. 

Graphical 

[1] 

[2] 

7. 

Accuracy 

[1] 

[23 

8. 

Credibility 

[1] 

[2] 

9. 

Reliability 

[1] 

[23 

10. 

Filtered/Processed 

[1] 

[2] 

11. 

Interpreted 

[1] 

[2] 

12. 

Relevance 

[13 

[2] 

13. 

[1] 

[2] 

14. 

11] 

[2] 

15. 

11] 

[2] 

Moderately 

Important 

Extremely 

Important 

[3] 

[4] 

15] 

[6] 

[7] 

E3] 

[4] 

15] 

[6] 

[7] 

[3] 

t4] 

15] 

[6] 

[7] 

[3] 

t4] 

15] 

[6] 

[7] 

[3] 

[4] 

15] 

[6] 

[7] 

[3] 

[4] 

15] 

[6] 

17] 

[3] 

[4] 

15] 

[6] 

[7] 

[3] 

[4] 

15] 

[6] 

[7] 

[3] 

[4] 

15] 

16] 

[7] 

[3] 

[4] 

15] 

[6] 

[7] 

[3] 

[4] 

15] 

[6] 

[7] 

[3] 

[4] 

15] 

[6] 

[7] 

[3] 

[4] 

[5] 

[6] 

[7] 

[3] 

[4] 

15] 

[6] 

17] 

[3] 

[4] 

15] 

[6] 

[7] 
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Appendix  B 

Decision  Conference  Read- Ahead  Package 
EXTERNAL  FACTORS 

Our  study  is  focusing  on  the  utility  of  outputs  from  military  intelligence.  By  outputs,  we 
mean  anything  that  operations  receives  from  intelligence,  including  formal  reports, 
briefings,  informal  conversations,  etc. 

Based  on  our  initial  research,  we  have  concluded  that  the  specific  utility  assigned  to 
the  information  contained  in  a  particular  intelligence  output  is  influenced  somewhat  by 
the  situational  context.  For  that  reason,  we  are  concerned  with  determining  what 
factors  define  the  context  for  the  individual  judging  the  utility/relevance  of  information 
contained  in  an  output. 

Six  external  (i.e.,  situational)  factors  have  been  identified  from  our  research.  We 
would  like  your  opinion  at  the  working  session  as  to  whether  or  not  the  list  of  factors 
and  levels  is  complete.  After  you  have  reviewed  all  the  factors,  please  consider 
whether  or  not  you  think  there  are  additional  factors  that  we  have  overlooked.  If  there 
are  additional  external  factors  that  impact  on  the  assessment  of  intelligence  output 
utility  please  point  these  factors  out  at  the  working  session. 


ExternaLFactQLl ;  ieyfii  of  .Conflict 

Level  of  conflict  of  the  situation  in  which  the  individual  is  assessing  utility. 

Levels  of  this  factor  include: 

Peace/Noncombat 

Terrorism 

Contingency  Operations 
Low-Intensity  Conflict 
Mid-Intensity  Conflict 
High-Intensity  Conflict 

1 .  Do  you  think  that  level  of  conflict  may  affect  the  utility/value/relevance  assessment  of 
information  contained  in  an  intelligence  output? 

2.  If  yes  to  question  1 ,  do  you  think  that  the  level  of  conflict  factor  is  adequately 
described  by  the  levels  as  stated?  Are  any  of  the  levels  unnecessary?  Should 
additional  levels  be  added?  Should  some  levels  be  combined?  Please  offer  specific 
suggestions  for  modifications. 
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Staff  position  of  the  individual  making  the  utility  assessment  of  an  intelligence  output. 


Levels  of  this  factor  include: 

G1 

G2 

G3 

G4 

G5 

CDR 

FSC 

Chemical  Officer 
Intrastaff  (G3Plans,  G30ps) 

1 .  Do  you  think  that  user  role  has  an  impact  on  the  assessment  of  the 
utility/value/relevance  of  information  contained  in  an  intelligence  output? 

2.  If  yes,  should  additional  users  be  included?  (EWO,  other  echelons  above,  below, 
adjacent) 


External  Factor  3:  User  Echelon 
Echelon  of  the  individual  making  the  utility  assessment. 
Levels  of  Factor: 


National 

U  &  S  Command 

Theater 

Corps 

Division 

BDE 

BN 


1.  Do  you  think  that  a  user's  echelon  impacts  the  utility/value/relevance  assessment  of 
information  contained  in  an  intelligence  output? 

2.  If  yes  to  question  1 ,  are  all  the  levels  of  the  factor  necessary?  If  no,  are  more 
needed  or  fewer?  Please  offer  specific  suggestions  for  modifications. 
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Operational  state  is  defined  as  conditions  of  readiness  or  action. 

Levels  of  Factor: 

Status  Quo  (continuation  of  routine  activities) 

Pre-Hostility  (DefCon  2  or  lower) 

Hostility  (Defcon  1  or  combat) 

Post-Hostility  (activities  associated  with  post-combat,  battle,  campaign,  or 
contingency  operations) 

1.  Do  you  think  that  operational  states  impact  the  assessment  of  utility/value/relevance 
of  information  contained  in  an  intelligence  output? 

2.  If  yes  to  question  1 ,  are  all  the  levels  of  the  factor  necessary?  If  no,  are  more 
needed  or  fewer?  Please  offer  specific  suggestions  for  modifications. 


Functional  application  governs  the  way  the  information  content  of  an  intelligence 
output  will  be  applied  by  the  user. 

Levels  of  Factor: 

Data  Base/Record  Keeping.  The  maintenance  of  files  and  development  of 
historical  information  from  which  comparative  analysis  is  undertaken. 

Protection/Security.  Those  passive  and  active  measures  undertaken  to  provide 
operational  security. 

Planning.  All  activities  necessary  for  operations  preparation  for  the 

accomplishment  of  mission  related  to  maneuver,  fires,  combat  support,  and 
combat  service  support. 

Execution.  All  activities  necessary  for  accomplishing  the  mission  related  to 
maneuver,  fires,  combat  support,  and  combat  service  support. 

1 .  Do  you  think  that  functional  application  impacts  the  assessment  of 
utility/value/relevance  of  information  contained  in  an  intelligence  output? 

2.  If  yes,  are  all  of  the  identified  factor  levels  necessary?  If  no,  are  more  needed  or 
fewer?  Please  offer  specific  suggestions  for  modifications. 
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External  Factor  6:  Mission 

Mission  refers  to  the  mission  of  the  user's  unit. 

Levels  of  Factor 

Offensive 

Defensive 

Retrograde 

Special  Operations 

Contingency  Operations 

Joint  and  Combined  Operations 

1.  Do  you  think  that  mission  impacts  the  assessment  of  utility/value/relevance  of 
information  contained  in  an  intelligence  output? 

2.  If  yes,  are  the  identified  levels  adequate?  If  no,  are  more  needed  or  fewer?  Please 
offer  specific  suggestions  for  modifications. 
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MEASUREMENT  OF  INFORMATION  VALUE/RELEVANCE 


The  current  focus  of  our  investigation  into  the  utility  of  intelligence  output  is  concerned 
with  the  value/relevance  of  information  contained  in  a  military  intelligence  output.  We 
have  extracted  (from  numerous  sources)  a  large  number  of  information  items.  These 
information  items  have  been  organized  into  a  hierarchical  structure.  We  would  like  you 
to  review  both  the  information  items  and  the  manner  in  which  they  have  been 
organized  and  answer  the  following  questions: 

Are  there  information  items  missing  which  should  be  included?  If  so, 
please  specify  the  items  to  be  added. 

Is  the  organization  of  the  information  items  adequate?  If  not,  how 
should  the  information  be  organized? 

During  the  working  session  we  would  like  to  obtain  consensus  on  the  following:  (a) 
information  items,  (b)  organization  of  the  information  items,  and  (c)  external  factors 
needed  to  define  situational  context.  Once  consensus  has  been  obtained,  we  will  ask 
you  to  weight  the  information  items  in  order  to  obtain  importance  weights  that  indicate 
the  relative  value  of  different  types  of  information  for  different  users  under  different 
situational  contexts. 

We  have  provided  two  different  portrayals  of  the  information  item  organization  for  your 
convenience.  While  both  portrayals  contain  the  same  information,  one  is  diagrammed 
in  a  tree-like  structure,  the  other  is  in  outline  form.  The  tree-like  structure  of  the 
information  organization  provides  an  overview  of  the  two  highest  levels  with  the  lower 
levels  of  detail  provided  on  subsequent  pages.  The  outline  form,  on  the  other  hand, 
steps  completely  through  the  organization  proceeding  from  the  highest  level  through 
its  related  detail  before  moving  to  the  next  highest  level.  You  may  use  either  or  both 
portrayal  whichever  is  more  convenient  for  you. 


Strawman  Factora  for  Maaaurlng  Information  Valua/Ralavanca  of  Ml  Output 


LEVEL  1 


LEVEL  2 


I. 


If. 


III. 


Reference  to  | 

Relevance  H 

-  A. 

of  Information  j 

Description  of 

RjkttlafiAlH  -  — 

Area 

Description 

of  Enemy 

Situation 

b. 

-  E. 

Rafars  to  mission 

Refers  to  commander's  guidance 

Weather 

Terrain 

Other  characteristics  of  area 

Enemy  disposition  and  composition  described 
Strength  of  enemy  force  described 
Recent  and  present  significant  activities  described 
Pecularities  and  weaknesses  described 
Special  targets  identified 


IV. 


V. 


VI. 


Description 

-  A. 

of  Enemy 
Cspabllities 

b. 

In  terms  of 

n 

* 

| 

Other  factors  1 

C. 

Enemy  COAs  enumerated 
Changes  in  enemy  situation  summarized 
Changes  in  battlefield  area  identified 
Enemy  capabilities  summarized 
Enemy  COAs  analyzed  and  discussed 

Effects  of  intelligence  considerations  on  operations 
Effects  of  the  area  of  operations  on  friendly.  COAs 
Probable  enemy  COAs 
Enemy  vulnerabilities 

Gaps  in  information  (and  their  implications) 
Inconsistencies  with  previous  information  identified 
Specificity  of  geographic  locations 
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Level  1:  II.  Description  of  Battlsflsld  Arts 


LmL2 


Levs  I  3 


t-svl  4 


A.  Weather 


B.  Terrain 


1 .  Describes  existing  situation 

2.  Describes  effects  on  enemy  COAs 

3.  Describes  effects  on  friendly  COAs 

1.  Describes  existing  situation 
in  terms  of 

2.  Describe  effect  on  enemy  COAs 

3.  Describes  effect  on  friendly  COAs 


a.  Observation 
and  fire 

b.  Cover  and 
concealment 

c.  Obstacles 

d.  Key  terrain 

e.  Avenues  of 
Approach 


C.  Other 
Characteristics 
in  terms  of 


1.  Existing  situation:  economic  and  psychological 

2.  Effects  on  enemy  COAs 

3.  Effects  on  friendly  COAs 
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Level  1:  III.  Description  of  Enemy  Situation 


Level  2 


Level  3 


A  Enemy  disposition  and 
composition  described 
in  terms  of 


1.  Forward  trace 

2.  Unit  locations  (assuming  id  of  type  or  type  unknown) 

3.  Main  area  of  attack 

4.  Echelonment 

5.  Support  elements 

6.  Reserves 

7.  Staging  areas 

8.  Supply  lines 

9.  Supply  areas 
lOAir  corridors 


B.  Strength  of  enemy  force 
description 


1. 

2. 

3. 

4. 

5. 

6. 

7. 


Describe  strength  of  committed  forces 
Describe  strength  of  reinforcements 
Identify  resupply  rates 
Identify  critical  nodes 
Indicate  level  of  morale 

Describe  strength  of  air  forces  supporting  enemy 
ground  forces 

Describe  nuclear,  biological,  and  chemical 


C. 


Recent  and  present  significant 
activities  described  in  terms  of 


Phase  lines  on  enemy  avenues  of  approach 

Activities  in  rear  area 

Activities  in  front  area 

Activities  in  deep  area 

Relevant  activity  outside  area  of  influence 

Movements  (combat  maneuver,  arty,  airborne, 

logistics) 

Activities  organized  functionally  (air,  nuc,  chem,  etc.) 


D  Peculates  and  weaknesses 
described  in  terms  of 


1.  Personnel 

2.  Intelligence 

3.  Operations  (includes  an  estimate  of  combat 
effectiveness) 

4.  Logistics 

5.  Civil-military  operations 

6.  Personalities 


E.  Special  targets  identified 
in  terms  of 


1.  High  value 

2.  Atypical 

3.  Intelligence  assets 

4.  Potential  nuclear  targets 

5.  Enemy  concentrations 

6.  Prohibited  targets/areas 


Laval  1:  IV.  Daacrlptlon  of  Enamy  Capabilities 


LiyaL2 


Laval  3 


Lml-4 


A.  Enumeration  of  enemy  COAs 


B.  Summary  of  changes 
in  enemy  situation 
in  terms  of 


1.  Order  of  battle 

2.  Movements 

3.  Unusual  aetivities/unanticipated  events 

4.  Strengths/weaknesses 

5.  Disposition  of  units 

6.  Replacement,  rotation,  reinforcement,  or  unit 
resupply 

7.  Intelligence  activities 

8.  Advance  to  new  phase  line 

9.  Enemy  COA  change 
lO.Indicators  of  decision  points 


C.  Changes  in  battlefield 
area  in  terms  of 


1.  Weather 

2.  Terrain 


D.  Summary  of  enemy 
capabilities  in 
terms  of 


1.  General  — -I 


2.  Special/ 
Commander 
Selected 


a.  Concentration  of  forces 

b.  Preparations 

c.  Readiness  levels 

d.  Intelligence  collection 
priorities 

e.  Relative  combat  power 

f.  Air  in  support  of  ground  forces 

g.  Strengths/weaknesses  in 

logistic/technical  capabilities 

h.  Time/distance  factors  (in 
terms  of  enemv  projected/ 
closure  time  a  .  friendly 
reaction  time) 

i.  Capability  to  hit  high  value 
friendly  locations 

j.  Capability  to  influence 
friendly  scheme  of  maneuver 


a.  NBC.  ADA 

b.  Airmobile,  airborne 

c.  Engineer 


E.  Analysis  and  discussion  of  enemy  COAs 
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Laval  1:  V.  Conclusion*  In  Tarm*  of 


Laval  2  Laval  3 


A.  Effects  of  intelligence 

considerations  on  operations 


8.  Effects  of  the  area  of  operations 
on  friendly  COAs 

.  1.  Concentration  of  forces 

C.  Probable  enemy  COAs  in  terms  — - - - - -  2.  Location  of  main  attack 

of  _  3.  Type  of  units  and  strength 

D.  Enemy  vulnerabilities 
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Appendix  C 


Military  intelligence  Product  Survey 

Purpose:  The  purpose  of  this  survey  is  to  obtain  the  opinion  of  operational  personnel 
with  regard  to  the  utility  of  military  intelligence  products.  The  enclosed  forms  will  ask 
you  about  your  familiarity  with  various  products  and  how  frequently  you  use  the 
products.  In  addition,  you  will  be  asked  your  opinion  about  product  adequacy  and 
possible  ways  to  improve  the  products.  Brief  instructions  are  contained  at  the  top  of 
each  survey  form;  please  read  these  before  completing  each  form. 

About  the  Survey:  By  intelligence  product  we  are  referring  to  outputs  of  military 
intelligence;  included  in  the  list  of  products  are  those  you  would  receive  in  written  form 
as  well  as  briefings,  including  formal  (e.g.,  commander's  evening  brief)  and  informal 
(e.g.,  conversations).  The  list  of  intelligence  products  contains  products  specified  by 
intelligence  doctrine  (e.g.,  INTSUM)  as  well  as  intelligence  outputs  that  are  not  formally 
recognized  as  products.  The  list  contains  some  outputs  of  the  tPB  process  (e.g., 
named  areas  of  interest)  and  intelligence  products  that  direct  the  intelligence  collection 
effort  (e.g.,  collection  priorities).  The  list  of  intelligence  products  does  not  contain  all 
possible  intelligence  products;  for  this  reason  spaces  are  provided  on  the  survey  forms 
for  you  to  add  intelligence  products  that  you  believe  should  be  included  in  the  list. 

You r  responses  on  these  forms  will  remain  strictly  confidential.  We  greatly  appreciate 
your  cooperation  in  this  study.  If  you  are  interested  in  obtaining  a  summary  of  the  study 
findings,  please  give  your  name  and  mailing  address  to  one  of  the  interviewers. 


Demographic  Information:  Please  complete  the  following. 

Rank _ 

Primary  Specialty _ 

Alternate  Specialty _ 

Have  you  served  on  an  intelligence  staff  or  in  an  operational  intelligence  unit  in  the 
past  five  years?  YES _  NO _ 


Please  identify  your  division  level  operational  staff  experience. 


PAGE  2:  INTELLIGENCE  PRODUCT  FAMILIARITY 

INSTRUCTIONS:  This  form  is  concerned  with  determining  how  well  acquainted  you  are  with  various  intelligence 
products.  For  each  product,  please  check  the  box  that  best  indicates  the  extent  of  your  familiarity  with  a 
particular  product.  For  example,  H  you  have  never  heard  of  an  INTSUM,  you  would  check  box  *1  on  the  line 
corresponding  to  an  INTSUM;  if  you  have  extensive  knowledge  of  an  INTSUM  you  would  check  box  #5,  and  so  on. 
You  may  add  products  in  the  spaces  provided  at  the  end  of  the  list.  Please  check  only  one  box  per  product. 


INTELLIGENCE  PRODUCT 

NOT  AT  ALL 
FAMILIAR 
WITH  IT 

SLIGHTLY 
FAMILIAR 
WITH  IT 

MODERATELY 
FAMILIAR 
WITH  IT 

VEFY 
FAMILIAR 
WITH  IT 

EXTREMELY 
FAMILIAR 
WITH  IT 

Intelligence  Summary  (INTSUM) 

m 

12] 

[3] 

(4] 

[5] 

Periodic  Intelligence  Report 
(PERINTREP) 

m 

(21 

[3] 

Ml 

[5] 

Supplementary  Intelligence  Report 
(SUPINTREP) 

m 

[2] 

[3] 

[4] 

[5] 

Intelligence  Annex 

m 

[2] 

13] 

[4] 

[5] 

Intelligence  Estimate 

m 

[2] 

[3] 

[4] 

[5] 

Intelligence  Report  (INTREP) 

Ml 

(2] 

[3] 

[4] 

[5] 

Situation  Report 

Ml 

[2] 

[3] 

[4] 

(5] 

Imagery  Reports  (such  as 
RECCEEXREP,  IPIR,  SUPIR) 

Ml 

12] 

[3] 

[4] 

[5] 

Spot  Report 

(11 

(2] 

[3] 

(4] 

[5] 

Responses  to  Information  Requests 

Ml 

12] 

[3] 

[4] 

(5] 

Formal  Briefing 

Ml 

12] 

[3] 

[4] 

[5] 

Informal  Briefing 

[i  1 

[2] 

[3] 

[4] 

[5] 

Collection  Priorities 

Ml 

[2] 

13] 

(4] 

(5] 

Named  Areas  of  Interest  (NAIs) 

[i] 

[2] 

(3] 

[4] 

[5] 

Target  Areas  of  Interest  (TAIs) 

Ml 

[2] 

(3] 

[41 

(5] 

Target  List 

[i] 

[2] 

13] 

(4] 

(5] 

Target  Nomination 

[i] 

[2] 

[3] 

(4] 

[5] 

Target  Priority  List 

IM 

[2] 

13] 

[4] 

[5] 

Other  Products  (please  list) 

Ml 

[2] 

(31 

(41 

(5) 

(il 

(2] 

(3] 

(4] 

[5] 

Ml 


121 

C-2 


131 


141 


151 


PAGE  3:  FREQUENCY  OF  NTELUGENCE  PRODUCT  USE 

INSTRUCTIONS:  This  form  is  concerned  with  determining  what  you  do  with  various  intelligence  products  when 
you  receive  them.  You  may  add  products  in  the  spaces  provided  at  the  end  of  the  list.  Check  as  many  boxes 
as  you  need  to  describe  how  frequently  you  use  each  product. 


INTELLIGENCE  PRODUCT 

DON'T  USE 
PRODUCT 

FILE  PRODUCT 
FOR  LATER  USE 

USE PRODUCT 
CNCE 

USE  PRODUCT 
REPEATEDLY 

USE  PRODUCT 
RIGHT  AWAY 

Intelligence  Summary  (INTSUM) 

M 

n 

[] 

M 

n 

Periodic  Intelligence  Report 
(PERINTREP) 

t  1 

[  1 

[  1 

[  1 

Supplementary  intelligence  Report 
(SUPINTREP) 

( 1 

[  1 

[  1 

[  1 

I  1 

Intelligence  Annex 

[  I 

t  1 

I  1 

[  1 

I  1 

Intelligence  Estimate 

[  1 

[  1 

(  1 

[  ] 

[  1 

Intelligence  Report  (INTREP) 

[  1 

[  ] 

[  1 

[  ] 

l  1 

Situation  Report 

[  ] 

[  } 

[  1 

[  1 

I  1 

Imagery  Reports  (such  as 
RECCEEXREP,  IPIR,  SUPIR) 

[  1 

[  1 

[  1 

[  1 

l  1 

Spot  Report 

I  ] 

[  ] 

t  1 

[  ] 

I  1 

Responses  to  Information  Requests 

l  1 

t  1 

[  ] 

I  1 

Formal  Briefings 

l  1 

[  1 

[  1 

[  1 

[  1 

Informal  Briefings 

I  1 

(  1 

I  1 

[  1 

l  1 

Collection  Priorities 

[  ] 

I  J 

[  1 

[  1 

[  1 

Named  Areas  of  Interest  (NAIs) 

[  1 

[] 

[  1 

1 1 

[  1 

Target  Areas  of  Interest  (TAIs) 

[  ] 

[  ] 

[  1 

[  ] 

[  1 

Target  List 

[  } 

I  1 

I  1 

[  1 

[  1 

Target  Nomination 

[  1 

11 

[  1 

1 1 

[  1 

Target  Priority  List 

[  ] 

(  1 

[  1 

[  1 

[  1 

Other  Products  (please  list) 

[  1 

l  1 

[  I 

[  1 

[  1 

[  1 

( 1 

[  1 

1  1 

l  1 

[  1 

[  ] 

l  1 

03 


PAGE  4:  NTELUGENCE  PRODUCT  USEFULNESS 

INSTRUCTIONS:  This  form  asks  for  your  opinion  about  the  usefulness  of  various  intelligence  products.  From  your 
experience,  we  would  like  your  opinion  about  how  well  each  intelligence  product  listed  fits  your  requirements  or 
is  sufficient  to  meet  your  intelligence  needs.  Please  indicate  the  usefulness  of  each  product  by  checking  the  box 
that  corresponds  to  your  opinion  about  the  product.  Please  select  only  one  box  per  product.  You  may  add  to  this 
list  of  products  in  the  space  provided  at  the  end  of  the  list.  If  you  add  products,  please  indicate  your  opinion  about 
the  usefulness  of  them.  If  there  are  products  in  this  list  that  you  have  never  used,  please  do  not  rate  their 
usefulness,  leave  the  corresponding  line  blank. 


INTELLIGENCE  PRODUCT 

NOT 

USEFUL 

SLIGHTLY 

USEFUL 

MODERATELY 

USEFUL 

VERf 

USEFUL 

EXTREMELY 

USEFUL 

Intelligence  Summary  (INTSUM) 

[1] 

12] 

13] 

[4] 

15] 

Periodic  Intelligence  Report 
(PERINTREP) 

m 

12] 

13] 

14] 

[5] 

Supplementary  Intelligence  Report 
(SUPINTREP) 

m 

12] 

[3] 

(4] 

15] 

intelligence  Annex 

m 

12] 

[3] 

[4] 

15] 

Intelligence  Estimate 

m 

121 

[3] 

[4] 

[5] 

Intelligence  Report  (INTREP) 

m 

[2] 

[3] 

[4] 

[5] 

Situation  Report 

m 

[2] 

[3] 

[4] 

15] 

Imagery  Reports  (such  as 
RECCEEXREP,  IPIR,  SUPIR) 

m 

12] 

13] 

14] 

15] 

Spot  Report 

m 

12] 

[3] 

[4] 

15] 

Responses  to  Information  Requests 

m 

[2] 

[3] 

[4] 

15] 

Formal  Briefings 

m 

12] 

13] 

14] 

15] 

Informal  Briefings 

m 

{2] 

[3] 

[4] 

[5] 

Collection  Priorities 

jij 

[2] 

[3] 

[4] 

[5] 

Named  Areas  of  Interest  (NAIs) 

m 

{2] 

13] 

14] 

[5] 

Target  Areas  of  Interest  (TAIs) 

f  i  j 

12] 

[3] 

[4] 

15] 

Target  List 

m 

[2] 

[3] 

[4] 

15] 

Target  Nomination 

m 

[2] 

13] 

14] 

15] 

Target  Priority  List  " 

m 

12] 

[3] 

[4] 

15] 

Other  Products  (please  list) 

m 

12] 

13] 

[4] 

15] 

Ml 

12] 

13] 

14] 

15] 

Ml 

121 

131 

141 

151 

C-4 


PAGES:  INTELLIGENCE  PRODUCT  REASON  FOR  USE 

INSTRUCTIONS:  This  form  asks  you  to  characterize  the  reason(s)  why  you  find  a  particular  intelligence 
product  to  be  useful.  If  you  gave  a  product  a  usefulness  rating  other  than  not  useful,  please  characterize 
the  reason  for  your  usefulness  rating  below.  You  may  check  as  many  boxes  as  needed  to  describe  why  a 
product  is  useful  to  you.  You  may  add  products  in  the  spaces  provided.  If  you  add  products  please 
indicate  your  reason(s)  for  using  the  products.  Please  do  not  rate  any  products  you  rated  as  not  useful. 


0M.Y90URCE  BEST  SOURCE 

EASY  TO 

MOST  TIMELY  OTTER 

AVAILABLE 

AVAILABLE 

USE 

INFORMATION  (specify) 

N7RDGENCE  PRODUCT 

Intelligence  Summary  (INTSUM) 
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l  1 

[  1 

[  1 

Periodic  Intelligence  Report 

[  1 

[  ] 

[  ] 

[  ] 

(PERINTREP) 

Supplementary  Intelligence  Report 

(  1 

[  ) 

l  1 

[  1 

(SUPINTREP) 

Intelligence  Annex 

[  I 

[  ] 

[  J 

[  ] 

Intelligence  Estimate 

[  ] 

l  1 

I  1 

[  1 

Intelligence  Report  (INTREP) 
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[  1 

[  1 

[  1 

Situation  Report 

[  1 

(  ] 

[  1 

[  1 

Imagery  Reports  (such  as 

l  1 

[  1 

[  1 

[  1 

RECCEEXREP,  IPIR,  SUPIR) 

Spot  Report 

I  1 

[  ] 

[  ] 

[  1 

Responses  to  Information  Requests 

I  1 

(  1 

[  1 

l  1 

Formal  Briefings 

I  1 

[  1 

[  ] 

l  1 

Informal  Briefings 

[  ) 

[  ] 

I  ] 

[  ] 

Collection  Priorities 

I  ] 

[  1 

[  ] 

[  1 

Named  Areas  of  Interest  (NAIs) 

{  ] 

[  1 

[  ) 

l  ] 

Target  Areas  of  Interest  (TAIs) 

[  ] 

I  1 

I  ] 

Target  List 

[  1 

I  1 

[  ] 

[  1 

Target  Nomination 

[  ] 

l  ] 

I  } 

l  J 

Target  Priority  fcist 

[  1 

[  ] 

[  ] 

[  ] 

Other  Products  (please  list) 

(  ] 

I  } 

[  ) 

l  ) 

[  ] 

l  ) 

[  1 

[  ] 

f  1 

r  i 

f  1 

r  i 
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PAGE  6:  RECOMMENDED  MODIFICATIONS  TO  INTELLIGENCE  PRODUCTS 
INSTRUCTIONS:  This  form  is  provided  for  you  lo  list  specific  recommendations  concerning  ways  to  improve 
products.  Please  list  the  product  in  the  space  provided  and  your  recommended  modifications  in  the  adjacent  space. 


NTELUGENCE  PRODUCT 


RECOMVENDED  MODIFICATIONS 


PAGE  7 


1.  In  answering  the  preceding  questions  about  intelligence  products,  did  you  think  of 
experiences  you  had  in  a  specific  command  or  unit  assignment.  If  yes,  please  specify 
the  command  or  unit  assignment  and  its  location. 


2.  When  making  your  responses  to  the  questions  about  intelligence  products,  did  you 
make  any  assumptions  about  the  accuracy  or  timeliness  of  the  intelligence,  or  the 
credibility  of  the  analyst,  etc.?  If  so,  please  explain. 


3.  Are  there  any  questions  that  you  feel  should  have  been  asked  that  were  not  asked? 
If  so,  what? 


4.  Since  this  study  is  focusing  on  the  operational  perspective  of  military  intelligence 
products,  is  there  anything  you  would  like  to  point  out  to  the  researchers? 


(If  you  require  additional  space  in  answering  the  questions  below,  please  continue 
answers  on  the  back  of  this  page.) 
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Appendix  D 


Information  Types 

Importance  in  Peacetime  and  Active  Hostilities 


Peacetime 

Active  Hostilities 

Information  Types 

Mean 

Standard 

Deviation 

Mean 

Standard 

Deviation 

Enemy  Capabilities 

5.75 

0.95 

7.00 

0 

Unit  Type 

6.00 

1.41 

7.00 

0 

Size 

5.75 

1.50 

7.00 

0 

Reinforcements 

4.75 

1.26 

6.50 

0.58 

Intelligence  Efforts 

4.00 

1.83 

5.00 

2.16 

Leadership 

4.40 

0.89 

5.20 

0.84 

Morale 

4.20 

1.10 

4.80 

1.10 

Weaknesses/Vulnerabilities 

5.20 

1.48 

6.40 

0.89 

Location  of  Enemy 

5.20 

1.79 

6.80 

0.45 

Enemy  Movement/Activity 

4.60 

2.51 

6.80 

0.45 

Enemy  Order  of  Battle 

5.80 

1.30 

6.80 

0.45 

Changes  in  Enemy  Posture 

5.00 

2.70 

6.60 

0.89 

Political  Climate 

4.50 

2.08 

3.60 

1.30 

Terrain 

5.60 

1.52 

6.40  . 

0.89 

Weather 

5.25 

1.5 

6.40 

0.89 

Rating  Scale:  1  to  7 
1  =  Not  important  at  all 
4  =  Moderately  important 
7  =  Extremely  Important 
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Output  Characteristics 

Importance  in  Peacetime  and  Active  Hostilities 


Peacetime 

Active  Hostilities 

Output  Characteristics 

Mean 

Mean 

BESSiBui 

Timeliness 

5.00 

2.12 

7.00 

0 

Currency 

5.80 

1.30 

7.00 

0 

Completeness 

5.75 

1.26 

5.40 

2.50 

Conciseness 

4.80 

0.84 

5.40 

0.89 

Clarity 

5.20 

1.10 

5.60 

1.14 

Graphical 

3.8 

0.84 

5.00 

1.20 

Accuracy 

6.20 

0.84 

6.80 

0.45 

Credibility 

5.80 

0.84 

6.40 

0.55 

Reliability 

5.80 

1.50 

6.40 

0.55 

Filtered/Process 

5.25 

1.50 

5.75 

0.96 

Interpreted 

5.00 

1.15 

5.75 

0.50 

Relevance 

4.80 

1.92 

5.60 

2.07 

Rating  Scale:  1  to  7 
1  =  Not  important  at  all 
4  =  Moderately  important 
7  =  Extremely  Important 
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Appendix  E 


GLOSSARY  OF  TERMS 

The  following  definitions  are  modifications  of  dictionary  definitions.  The 
objective  of  the  modification  of  the  definitions  was  to  capture  an  operational 
perspective  in  the  definitions  to  be  used  in  the  MI  unit  research  work. 

Accuracy:  The  condition  or  quality  of  representing  reality  about  enemy  forces, 
enemy  capabilities,  enemy  intentions,  weather,  or  terrain.  For  example,  the 
degree  to  which  something  is  where  it  is  said  to  be. 

Gear.  Information  or  finished  intelligence  about  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather,  or  terrain  which  is  free  from  obscurity  or 
ambiguity,  and  is  easily  understood  and  unmistakable. 

Complete:  Information  or  finished  intelligence  about  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather,  or  terrain  which  has  all  the  necessary 
parts  or  elements,  is  fully  developed  and  does  not  carry  the  implication  that 
something  is  missing. 

Concise:  Information  or  finished  intelligence  about  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather,  or  terrain  which  covers  much  in  a  few 
words,  is  succinct  and  free  from  elaboration  and  superfluous  detail.  See  "detail" 
and  "summary"  for  related  concepts. 
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Conclusion:  A  reasoned  judgment,  such  as  the  enemy  most  likely  courses  of 
action,  accepted  as  a  possible  outcome  with  regard  to  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather,  or  terrain. 

Continuity:  Information  or  finished  intelligence  about  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather,  or  terrain  which  represents  a  succession 
of  data  without  essential  change.  See  "gap"  for  related  concept. 

Credibility:  The  demonstrated  quality,  over  time,  of  presenting  finished 
intelligence  about  enemy  forces,  enemy  capabilities,  enemy  intentions,  weather, 
or  terrain  which  is  accepted  by  users  as  being,  more  often  than  not,  correct. 

Detail:  Information  or  finished  intelligence  about  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather,  or  terrain  which  is  an  extended  treatment 
or  exhaustive,  thorough  and  comprehensive,  but  focused  on  a  specific  operational 
purpose  or  use.  See  "concise"  and  "summary"  for  related  concepts. 

Filtered:  Information  or  finished  intelligence  about  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather,  or  terrain  which  has  been  pre-screened  to 
remove  unnecessary  redundancy  or  spurious  facts  or  data. 

Finished  Intelligence:  The  result  of  evaluation,  analysis  or  fusion  of  facts  or  data 
withr  new  facts  or  data  pertaining  to  enemy  forces,  enemy  capabilities,  enemy 
intentions,  weather,  or  terrain.  See  "information"  for  contrasting  concept. 
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Gap:  Information  or  finished  intelligence  about  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather,  or  terrain  which  reflects  a  break  in 
continuity.  See  "continuity''  for  related  concept 

Graphic:  Information  or  finished  intelligence  about  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather,  or  terrain  which  is  portrayed  as  a  picture, 
map,  overlay,  chart,  table,  or  graph. 

Hypothesis:  A  proposition  or  set  of  propositions,  such  as  potential  enemy 
courses  of  action,  set  forth  as  an  explanation  or  projection  of  what  may  occur 
with  regard  to  enemy  forces,  enemy  capabilities,  enemy  intentions,  weather,  or 
terrain. 

Information:  Facts  or  data  pertaining  to  the  enemy  forces,  enemy  capabilties, 
enemy  intentions,  weather,  or  terrain.  See  "finished  intelligence”  for  contrasting 
concept. 

Precise:  Information  or  finished  intelligence  about  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather  or  terrain  which  is  definitely  or  strictly 
stated,  defined  or  fixed;  minutely  exact 

Perishability:  Information  or  finished  intelligence  about  enemy  faces,  enemy 
capabilities,  enemy  intentions,  weather,  or  tenuin  which  by  its  very  nature 
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deteriorates  rapidly  in  terms  of  permiting  reasonable  operational  response.  See 
"timeliness"  for  a  related  concept. 


Processed:  Information  or  finished  intelligence  about  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather,  or  terrain  which  has  been  synthesized 
with  other  information  or  finished  intelligence. 

Relevance:  The  degree  to  which  information  or  finished  intelligence  about  enemy 
forces,  enemy  capabilities,  enemy  intentions,  weather,  or  terrain  applies  to  the 
mission  and  area  of  operations  and  satisfies  the  needs  of  the  user. 

Reliablitv:  The  quality  or  state  of  being  dependable  in  the  portrayal  of 
information  or  finished  intelligence  about  enemy  forces,  enemy  capabilities, 
enemy  intentions,  weather,  or  terrain. 

Specific:  Information  or  finished  intelligence  about  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather,  or  terrain  which  is  peculiarly  adapted  to 
an  operational  purpose  or  use,  such  as  planning,  targetting,  or  maneuver. 

Summary:  Information  or  finished  intelligence  about  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather,  or  terrain  which  is  comprehensive  and 
covers  the  main  points  succinctly.  See  "concise"  and  "detail"  for  related 
concepts. 
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The  availability  of  information  or  finished  intelligence  about  enemy 


forces,  enemy  capabilities,  enemy  intentions,  weather,  or  terrain  in  terms  of 
whether  or  not  it  permits  reasonable  operational  response.  See  "perishability"  for 
a  related  concept 

Unusualness:  Information  or  finished  intelligence  about  enemy  forces,  enemy 
capabilities,  enemy  intentions,  weather,  or  terrain  which  is  uncommon,  rare,  or 
unanticipated. 
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Appendix  F 


Intelligence  Product  Familiarity:  Operations  User  Group 


Intelligence  Product 

Mean 

Standard 

Deviation 

Minimum 

Maximum 

Median 

Intelligence  Summary 

4.27 

.47 

4.0 

5.0 

4.0 

Periodic  Intelligence  Report 

2.91 

1.04 

2.0 

5.0 

3.0 

Supplementary  Intelligence  Report 

2.27 

.90 

1.0 

3.0 

3.0 

Intelligence  Annex 

4.36 

.81 

3.0 

5.0 

5.0 

Intelligence  Estimate 

4.45 

.69 

3.0 

5.0 

5.0 

Intelligence  Report 

3.55 

.93 

2.0 

5.0 

3.0 

Situation  Report 

4.27 

.79 

3.0 

5.0 

4.0 

Imagery  Reports 

2.18 

.75 

1.0 

4.0 

2.0 

Spot  Report 

4.45 

.69 

3.0 

5.0 

5.0 

Responses  to  Information  Request: 

3.09 

1.14 

1.0 

5.0 

3.0 

Formal  Briefing 

4.73 

.47 

4.0 

5.0 

5.0 

Informal  Briefing 

4.73 

.47 

4.0 

5.0 

5.0 

Collection  Priorities 

4.00 

1.10 

2.0 

5.0 

4.0 

Named  Areas  of  Interest  (NAIs) 

4.09 

.83 

3.0 

5.0 

4.0 

Target  Areas  of  Interest  (TAIs) 

4.09 

.83 

3.0 

.  5.0 

4.0 

Target  List 

4.36 

.81 

3.0 

5.0 

5.0 

Target  Nomination 

4.00 

1.0 

2.0 

5.0 

4.0 

Target  Priority  List 

4.09 

1.04 

2.0 

5.0 

BSI 

Intelligence  Product  Familiarity:  Non-Operations  User  Group 


Intelligence  Product 

Mean 

Standard 

Deviation 

Minimum 

Maximum 

Median 

Intelligence  Summary 

4.25 

.45 

4.0 

5.0 

4.0 

Periodic  Intelligence  Report 

3.83 

.83 

3.0 

5.0 

4.0 

Supplementary  Intelligence  Report 

2.83 

1.47 

1.0 

5.0 

3.0 

Intelligence  Annex 

4.33 

.49 

4.0 

5.0 

4.0 

Intelligence  Estimate 

4.17 

.72 

3.0 

5.0 

4.0 

Intelligence  Report 

3.83 

.94 

2.0 

5.0 

4.0 

Situation  Report 

4.58 

.51 

4.0 

5.0 

5.0 

Imagery  Reports 

2.50 

1.45 

1.0 

5.0 

2.0 

Spot  Report 

4.58 

.51 

4.0 

5.0 

5.0 

Responses  to  Information  Request 

2.63 

1.62 

1.0 

5.0 

2.5 

Formal  Briefing 

4.17 

1.03 

2.0 

5.0 

4.5 

Informal  Briefing 

4.42 

.67 

3.0 

5.0 

4.5 

Collection  Priorities 

3.42 

1.24 

1.0 

5.0 

3.0 

Named  Areas  of  Interest  (NAIs) 

3.42 

1.51 

1.0 

5.0 

3.5 

Target  Areas  of  Interest  (TAIs) 

3.50 

1.57 

1.0 

5.0 

3.5 

Target  List 

3.67 

1.50 

1.0 

4.0 

Target  Nomination 

3.67 

1.50 

1.0 

5.0 

4.0 

Target  Priority  List 

3.50 

1.51 

1.0 

5.0 

4.0 

Intelligence  Product  Frequency  of  Use 


Intelligence  Product 

Response  Percentage 

Don't 

Use 

File 

Use 

Once 

ESSsiH 

Use 

Right  Away 

Intelligence  Summary 

K39 

N.O. 

K39 

N.O. 

ESS 

[SHI 

E51 

N.O. 

KH 

ISRI 

0 

0 

0 

13 

21 

25 

57 

38 

21 

25 

Periodic  Intelligence  Report 

8 

D 

0 

n 

39 

50 

23 

21 

31 

14 

Supplementary  Intelligence  Report 

25 

27 

0 

oj 

63 

45 

13 

9 

0 

18 

Intelligence  Annex 

0 

n 

n 

n 

20 

B 

47 

71 

27 

B 

Intelligence  Estimate 

0 

D 

n 

B 

20 

13 

40 

47 

33 

27 

Intelligence  Report 

0 

8 

0 

8 

50 

39 

33 

31 

17 

15 

Situation  Report 

0 

0 

6 

B 

19 

21 

38 

21 

38 

50 

Imagery  Reports 

14 

46 

29 

0 

14 

36 

29 

9 

14 

9 

Spot  Report 

6 

0 

13 

14 

31 

36 

13 

B 

38 

43 

Responses  to  Info  Requests 

27 

36 

9 

0 

27 

18 

9 

18 

27 

27 

Formal  Briefing 

0 

D 

0 

B 

23 

29 

54 

29 

23 

29 

Informal  Briefing 

0 

0 

0 

B 

16 

29 

62 

29 

23 

36 

Collection  Priorities 

0 

15 

8 

15 

31 

31 

39 

31 

23 

8 

Named  Areas  of  Interest  (NAIs) 

8 

18 

8 

9 

15 

18 

54 

27 

15 

27 

Target  Areas  of  Interest  (TAIs) 

8 

9 

8 

0 

15 

36 

54 

27 

15 

27 

Target  List 

0 

8 

B 

15 

14 

15 

m 

38 

21 

23 

Target  Nomination 

0 

18 

8 

9 

23 

18 

46 

36 

23 

18 

Target  Priority  List 

8 

15 

0 

15 

25 

0 

50 

46 

17 

23 

Ops  =  Operations  Users 
N.O.  =  Non-Operations  Users 
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Intelligence  Product  Usefulness:  Operations  User  Group 


Standard 


Intelligence  Product 

Mean 

Deviation 

Minimum 

Maximum 

Median 

Intelligence  Summary 

4.09 

.83 

2.0 

5.0 

4.0 

Periodic  Intelligence  Report 

3.45 

1.04 

1.0 

5.0 

4.0 

Supplementary  Intelligence  Report 

2.75 

1.28 

1.0 

5.0 

3.0 

Intelligence  Annex 

4.00 

1.00 

2.0 

5.0 

4.0 

Intelligence  Estimate 

4.30 

.95 

2.0 

5.0 

4.5 

Intelligence  Report 

3.55 

.69 

2.0 

4.0 

4.0 

Situation  Report 

3.91 

1.04 

2.0 

5.0 

4.0 

Imagery  Reports 

3.10 

1.29 

1.0 

5.0 

3.5 

Spot  Report 

3.73 

1.35 

2.0 

5.0 

4.0 

Responses  to  Information  Request; 

3.10 

1.10 

1.0 

4.0 

3.5 

Formal  Briefing 

3.27 

.65 

2.0 

4.0 

3.0 

Informal  Briefing 

4.00 

1.00 

2.0 

5.0 

4.0 

Collection  Priorities 

3.64 

1.03 

2.0 

5.0 

3.0 

Named  Areas  of  Interest  (NAIs) 

4.00 

.77 

3.0 

5.0 

4.0 

Target  Areas  of  Interest  (TAIs) 

4.18 

.75 

3.0 

5.0 

4.0 

Target  List 

3.91 

1.04 

2.0 

5.0 

4.0 

Target  Nomination 

3.82 

.87 

3.0 

5.0 

4.0 

Target  Priority  list 

3.82 

.75 

3.0 

5.0 

4.0 

Intelligence  Product  Usefulness:  Non*Operations  User  Group 


Intelligence  Product 

Mean 

Standard 

Deviation 

Minimum 

Maximum 

Median 

Intelligence  Summary 

4.00 

.85 

2.0 

5.0 

4.0 

Periodic  Intelligence  Report 

3.42 

1.00 

2.0 

5.0 

3.0 

Supplementary  Intelligence  Report 

3.00 

.71 

2.0 

4.0 

3.0 

Intelligence  Annex 

3.50 

.67 

3.0 

5.0 

3.0 

Intelligence  Estimate 

4.00 

.60 

3.0 

5.0 

4.0 

Intelligence  Report 

3.83 

.83 

3.0 

5.0 

4.0 

Situation  Report 

4.42 

.67 

3.0 

5.0 

4.5 

Imagery  Reports 

2.78 

1.20 

1.0 

5.0 

3.0 

Spot  Report 

4.17 

.83 

3.0 

5.0 

4.0 

Responses  to  Information  Request! 

3.20 

1.40 

1.0 

5.0 

3.0 

Formal  Briefing 

3.42 

.90 

2.0 

5.0 

3.5 

Informal  Briefing 

3.92 

.90 

2.0 

5.0 

4.0 

Collection  Priorities 

2.91 

1.58 

1.0 

5.0 

3.0 

Named  Areas  of  Interest  (NAIs) 

3.30 

1.49 

1.0 

5.0 

3.5 

Target  Areas  of  Interest  (TAIs) 

3.30 

1.34 

1.0 

5.0 

3.5 

Target  List 

3.40 

1.26 

1.0 

• 

© 

3.5 

Target  Nomination 

3.00 

1.56 

1.0 

5.0 

3.0 

Target  Priority  List 

3.20 

1.55 

1.0 

5.0 

4.0 

Intelligence  Product  Reason  for  Use 


Intelligence  Product 

Response  Percentage 

Only  Source 

Best  Source 

Easy  to  Use 

Most  Timely 

Intelligence  Summary 

Ops 

N.O. 

Ops 

N.O. 

Ops 

N.O. 

!  Ops 

N.O. 

0 

30 

20 

60 

80 

10 

D 

0 

Periodic  Intelligence  Report 

0 

11 

20 

67 

40 

22 

40 

0 

Supplementary  Intelligence  Report 

33 

17 

33 

50 

33 

33 

0 

0 

Intelligence  Annex 

0 

18 

60 

46 

40 

27 

0 

9 

Intelligence  Estimate 

0 

18 

40 

46 

40 

27 

20 

9 

Intelligence  Report 

0 

20 

33 

30 

67 

40 

0 

10 

Situation  Report 

0 

0 

14 

15 

43 

15 

43 

69 

Imagery  Reports 

100 

38 

0 

25 

0 

25 

0 

12 

Spot  Report 

0 

0 

17 

15 

0 

23 

83 

62 

Responses  to  Info  Requests 

0 

14 

33 

29 

0 

14 

67 

43 

Formal  Briefing 

50 

29 

0 

14 

50 

29 

0 

29 

Informal  Briefing 

0 

0 

0 

9 

83 

36 

17 

55 

Collection  Priorities 

0 

50 

50 

33 

50 

17 

0 

0 

Named  Areas  of  Interest  (NAIs) 

0 

17 

50 

33 

17 

50 

33 

0 

Target  Areas  of  Interest  (TAIs) 

0 

0 

50 

50 

17 

50 

33 

0 

Target  List 

0 

33 

ES 

33 

100 

33 

0 

0 

Target  Nomination 

0 

33 

D 

50 

100 

17 

0 

0 

Target  Priority  List 

0 

33 

33 

33 

67 

33 

fl 

0 

Ops  =  Operations  Users 
N.O.  =  Non-Operations  Users 


Recommendations  for  Intelligence  Product  Improvement 


Intelligence  Product 

Recommendation 

Intelligence  Summary 

Improve  timeliness  or  reporting  cycle 

Increase  graphics 

Time  cycle  should  be  tied  to  intensity  of 
conflict 

Go  to  graphic  ENTSUM  format 

Periodic  Intelligence  Report 

Improve  timeliness 

Increase  graphics 

Supplementary  Intelligence  Report 

Improve  timeliness 

Increase  graphics 

Intelligence  Annex 

Shorten 

Use  matrix  format 

Improve  content 

Intelligence  Estimate 

Lengthen 

Eliminate  redundancy 

Increase  graphics 

Improve  content 

Intelligence  Report 

Improve  content 

Situation  Report 

Improve  content 

Imagery  Reports 

Improve  timeliness 

If  it's  more  than  2  horn  ^  old  -  it's  probably 
no  good 

Spot  Report 

Improve  content 

Salute  format  should  include  time  of 
occurence,  not  time  of  submission 

Responses  to  Information  Requests 

Improve  content 

Improve  timeliness 

Formal  Briefings 

Shorten 

Frequently  too  bogged  down  in  detail  (leave 
the  detail  for  the  INTSUM  and  INTREP) 

Informal  Briefings 

Improve  content 

Too  often  don't  answer  who,  what,  when, 
where,  how,  why  -  always  have  to  come  back 
and  ask  questions  (refers  to  G2/S2  tactical 
update) 

Collection  Priorities 

Explain  how  the  priorities  am  established  and 
how  they  are  satisfied 
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MI  Product  Usage  Survey: 
Comments  from  Participants 


"The  utility  of  any/all  intelligence  products  is  based  on  two  factors  and  two  factors 
only  (any  other  are  sub-factors  of  the  two):  two  factors  are  timeliness  and 
credibility.  Most  intelligence  reports  are  unfortunately  historical  in  nature  rather 
than  predictive  which  directly  relates  to  credibilty.  Intelligence  analysis  is  an  art  not 
a  science.  All  of  the  templates  in  the  world  are  useless  unless  the  analyst  has  an 
intuitive  grasp  of  how  they  relate  to  'ground  truth.'  We  must  capture  somehow  and 
quantify  what  tools  make  a  good  analyst.  Obviously,  an  analyst's  own  confidence 
and  the  how  of  that  confidence  must  be  captured.  We  must  teach  risk  versus  gain; 
we  must  teach  "how  to  think";  the  process  if  properly  taught  will  result  in  a  more 
timely  and  credible  product  Intelligence  reporting  is  not  rale  and  the  success  of 
intelligence  reporting  is  not  measured  by  pages  produced  but  rather  in  lives  saved  or 
lost.  Good  analysts  and  good  analysis  is  ngi  pontifi cation  but  a  useable  credible 
product  i.e.,  the  (enemy)  is  here  in  this  strength  and  he  is  going  to  do  this.  I  am  at 
a  loss  to  tell  you  how  we  get  from  where  we  are  now  to  where  we  need  to  be,  I 
simply  know  we  must  do  something.  Perhaps  we  need  to  design  a  course  in 
"Fundamentals  of  Analysis"  which  is  situational  and  equipment  independent:  one 
which  teaches  how  to  do  analysis.  Then  and  only  then  will  senior  intelligence  types 
realize  that  they  are  users  of  intelligence  as  well  as  producers  with  the  end  result  of 
timely  and  credible  intelligence  products."  Comments  from  a  division  G2. 
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"Intelligence  reports  should  go  farther  in  stating  or  estimating  the  impact  of  new 
information  as  it  relates  to  the  tactical  situation  and  the  capabilities/probable  courses 
of  action  available  to  the  enemy.  That  is  -  should  contain  preliminary  analysis  of 
the  impact  of  new  information  —  how  does  it  correlate  with  what  we  already  knew. 

Intelligence  estimates  are  generally  vague  and  deal  with  broad  brush  statements  of 
likely  enemy  courses  of  action.  Need  to  be  more  specific  at  the  risk  of  being 
wrong.  Answer  specially  —  that  based  upon  analysis  of  all  intelligence  available, 
here’s  what  the  enemy  is  likely  to  do:  who,  what,  when,  where,  how,  why.  It’s 
up  to  the  commander  to  determine  the  validity  of  the  estimate  in  order  to  make  his 
operational  decisions.  Granted,  if  you're  consistently  wrong  you'll  be  seeking 
employment  elsewhere.  But  don't  hedge  the  bets  with  broad  brush  statements. 

The  few  imagery  reports  I've  seen  were  so  old  as  to  be  stale  by  the  time  I’d  seen 
them.  Interested  in  target  locations  for  artillery  fire  or  close  air  support  If  it's 
more  than  2  hours  old  -  it's  probably  no  good. 

Spot  reports  -  the  Salute  format  is  very  good  but  as  a  rule  folks  submitting  Spot 
Reports  don't  include  sufficient  detail  for  me  to  be  able  to  use  the  information.  Big 
problem  is  with  location  accuracy  and  timeliness.  Recommend  that  Salute  include 
time  -  time  of  occurence,  not  time  of  submission. 

Formal  briefings  frequently  get  bogged  down  in  detail.  Not  interested  in  unit 
designation,  etc.  but  in  the  fact  that  there  are  X  number  of  regiments  of  this  type  at 


these  approximate  locations.  Leave  the  detail  to  the  INTSUM  and  INTREP.  Tell 
me  in  general  terms  whats  out  there  in  what  strength  -  what  their  strengths  and 
weaknesses  are,  and  most  importantly  ~  what  they're  likely  to  do. 

Informal  briefings  -  too  often  don’t  answer  who  what  when  where  how  why  - 
always  have  to  come  back  and  ask  questions  later.  In  this  I  refer  to  a  G2/S2  update 
in  tactical  situation  one-on-one. 

To  many  —  myself  included  -  the  whole  procedure  by  which  intelligence  is 
obtained  and  processed  is  vague,  mysterious,  and  therefore  suspect  Relationships 
between  MI  battalions/S2/G 2/etc  most  confusing. 

1.  My  interest  is  in  locating  targets  with  sufficient  accuracy  to  engage  with  artillery 
(+  100  meters).  G2/S2  needs  to  be  able  to  advise  on  reliability  and  accuracy  of 
source. 

2.  Timeliness  is  critical.  Old  reports  on  location  of  mobile  forces  are  worthless  if 
more  than  just  a  few  minutes  old. 

v 

3.  IVe  never  questioned  the  credibility  of  intelligence  analysts  -  they've  always 
proven  to  know  a  lot  mere  about  enemy  doctrine,  organization  for  combat 
capabilities,  current  situation  than  me.  Sometimes  I  has  disagreed  with  their 
conclusions  -  but  it's  ok  to  be  wrong  as  long  as  an  earnest  effort  was  made.  The 
biggest  problem  I'd  had  is  in  getting  the  analyst  to  commit  himself.  They  always 


stop  short  of  saying  this  is  what  I  believe  the  enemy  is  going  to  do."  Comments 


from  a  division  FS 


"Brevity  is  the  key  to  future  needs.  Short,  concise  pieces  of  very  useable 
information  that  can  be  transmitted  via  computers."  Comments  from  a  division  G3, 


"  Overall,  reduce  the  bulk,  size,  and  volume  of  (intelligence)  reports."  Comments 
from  a  division  G3. 


Regarding  assumptions  about  intelligence  —  "I  believe  everything  they  tell  me  — 
with  a  grain  of  salt." 

Experience  with  intelligence  -  "trying  to  get  information  from  intelligence  on 
terrorist  activity  was  like  pulling  teeth  from  a  duck  -  die  only  time  we  got  any 
information  was  usually  long  after  the  fact  -  this  did  not  allow  us  to  prepare  for 
contingencies.” 

Regarding  all  products:  "Intelligence  information  is  often  so  closely  guarded  by 
intelligence  personnel  as  to  make  it  useful  to  no  one.  The  information  tends  to  flow 
only  in  intelligence  channels,  to  the  point  that  die  user  (ops,  log,  etc.)  must  rely 
only  on  his  intuition  not  fact" 

"Something  needs  to  be  done  about  the  disseminadon  of  information  out  of 
intelligence  channels."  Comments  from  a  logistics  officer. 


F-ll 


Regarding  the  improvement  of  intelligence  products:  "Go  to  decision  support 
graphics  supported  by  a  short  written  matrix;  COA  sketches;  value  of  templates  as 
an  intelligence  product;  speed  TACFAX  type  dissemination  up  and  down  die 
echelons  and  lateral. 

'.Scope'  the  products  to  the  consumer  echelon  below  the  reporting  echelon.  i.e.,  a 
division  sends  raw  data  rapidly  down  to  bde  using  spot  reports  but  a  division  send 
a  history  report  to  corps.  Probably  should  not  even  send  it  information  to  bdes, 
but  probably  will  have  to."  Comments  from  an  intelligence  officer. 


"INTSUM.  Time  cycles  on  intelligence  input  must  be  tied  to  the  intensity  of 
conflict,  12  hours  or  24  hour  reports  in  high  intensity  may  be  outdated  before  they 
axe  received."  Comments  from  a  division  chemical  officer. 

"Generally,  the  most  significant  shortcoming  relative  to  intelligence  involves  the 


overwhelming  volume  of  information  their  officers  produce.  Suggest  commander 
receive  critical  information  only.  Good  start  point  is  CCIR  outlined  in  FC  101-55. 


